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EXECUTIVE SUMMARY 



To address concerns about the integrity of the process used by school districts for 
establishing eligibility for free and reduced-price meals in the National School Lunch Program 
(NSLP) and School Breakfast Program (SBP), the U.S. Department of Agriculture (USDA) 
sponsored the NSLP Application/Verification Pilot Projects. In particular, USDA asked districts 
around the country to voluntarily participate by testing ways of improving the process for 
certifying students as approved to receive free or reduced-price meals. They also commissioned 
an evaluation of two of these approaches — Up-Front Documentation and Graduated 
Verification — in 12 public school districts nationally. The evaluation examined the 
implementation of the pilots, assessed their costs, and estimated their impacts on a set of 
outcomes designed to measure the success of the school meal programs in providing free or 
reduced-price meals to their target population without providing benefits to ineligible students. 
This report focuses on impacts of the pilots on students’ participation in the NSLP. In particular, 
the report examines the impacts of Up-Front Documentation and Graduated Verification on the 
extent to which students get any school lunch, as well as the extent to which they buy free meals, 
reduced-price meals, or full-price (or paid) meals, both among all students and among those 
whose incomes make them eligible for free and/or reduced-price meals. 



What Did the Pilot Projects Attempt to Achieve? 

The overall objective of the pilot projects was to deter ineligible students from being 
approved for and getting free or reduced-price meals while at the same time avoiding raising 
barriers to certification and participation among students eligible for these benefits. Under 
federal guidelines, children living in families with incomes of 130 percent or less of the federal 
poverty level qualify for free meals, while those in families with income of between 130 and 185 
percent of poverty qualify for reduced-price meals. In addition, children receiving food stamp. 
Temporary Assistance for Needy Families (TANF), or Food Distribution Program on Indian 
Reservations (FDPIR) benefits are categorically eligible for free meals. 

Most children become certified for free meals by submitting an application on which they 
report their income and household size or food stamp/TANF/FDPIR case number. Under 
standard federal rules, no documentation of applicants’ income or benefit receipt is required at 
the time of application. In the verification process, however, districts must select a small sample 
of applications that have already been approved and collect income or benefit documentation 
from approved families in order to verify their eligibility for free or reduced-price meals. 

Some districts also establish eligibility for the NSLP using a process called direct 
certification, whereby students on food stamps/TANF/FDPIR can bypass the application process. 
Information from the food stamp or welfare offices in the states in which these districts are 
located is used to identify students on assistance, and these students can become certified for free 
meals without having to complete an application. 
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The pilots were designed to address a perceived problem that the number of children 
approved for free or reduced-price meals from families with incomes too high to qualify for the 
benefits they are receiving is large and perhaps growing. In particular, each of the pilot 
approaches intensifies the effort of districts to require students to document their income or 
benefit receipt in order to become certified or maintain their certification status. 

Nine districts included in the evaluation implemented Up-Front Documentation (UFD). 
Under this pilot, districts required all applicants for free or reduced-price meals to provide 
documentation, either of their income or receipt of public assistance, with the application. 
Benefits were not granted if the application did not include the required documentation. After 
the applications and documentation were reviewed and approved, the districts were not required 
to conduct any subsequent verifications. 

Three districts included in the evaluation implemented Graduated Verification (GV), under 
which the standard verification process was enhanced. Districts using GV first conducted the 
standard verification process by verifying a small sample of approved applications. Unlike the 
standard process, however, these districts conducted up to two additional rounds of verification. 
If at least 25 percent of the initially verified applications had their benefits reduced or terminated 
as a result of the verification process, districts were required to verify an additional 50 percent of 
remaining applications. Similarly, if 25 percent of these second-round applications resulted in 
benefit reduction/termination, districts were required to verify all remaining applications. 
Finally, students whose benefits were reduced/terminated as a result of verification were required 
to submit documentation with their application if they applied for benefits in the following year. 



Evaluation Design 

To estimate the impact of UFD and GV on participation, the evaluation used a comparison 
group design. For each pilot district, we selected a comparison district with similar 
characteristics during the pre-pilot period, and that was judged to be similar to the pilot district 
by program staff familiar with both districts. These comparison districts were used to establish 
benchmarks for measuring the impacts of UFD and GV. We estimated impacts by comparing 
average outcomes among students in the pilot districts with those in the comparison districts, 
controlling for preexisting differences in the observed characteristics of the two groups. 

Within the pilot and comparison districts, samples of students were selected that were 
representative of all the district’s students except those who were directly certified. Directly 
certified students were excluded from the sample because they were not subject to the pilot 
project rules and therefore not expected to be affected by the pilot procedures. Our analysis 
sample includes 1,950 students’ families in UFD pilot and comparison districts, and 1,070 
families in GV districts. 

Key information on sampled families was obtained by administering two surveys to the 
student’s parent or guardian, and by examining administrative data provided by the districts 
included in the evaluation. Initially, we conducted a telephone interview with the parent or 
guardian of each student in the sample, in which respondents were asked about experiences with 
and perceptions of the program — including the extent to which their child ate school lunches — 
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and about characteristics of their family. They were also asked a simple set of questions about 
their household size and income. Unless the respondent reported a very high income level, we 
then conducted an in-person interview with the family. The purpose of the in-person interview 
was to collect more detailed information about the household’s size and income. In particular, 
respondents were asked detailed questions about each source of income for each household 
member, and were also asked to retrieve documentation of income amounts. To obtain highly 
accurate information on the certification status of all sample members, we relied on 
administrative records provided by the pilot and comparison districts. 



The Impacts of Up-Front Documentation 

Overall, UFD led to a decrease in the proportion of all students getting a school lunch on a 
given day. The estimated impact of the pilot on the overall participation rate (as reported by 
parents) was -5.8 percentage points, and was statistically significant. This impact represented a 
reduction in participation of about 8 percent from the parent-reported rate of 72.6 in comparison 
districts. 

The overall decrease in participation brought about by UFD arose from a combination of a 
statistically significant decline in the free/reduced-price meal participation rate of 2.8 percentage 
points (or 15 percent) and a decline in the paid participation rate of 3.3 percentage points (6 
percent) that was not statistically significant. 



Estimated Impact of Up-Front Documentation on Rates of 
NSLP Participation Among Non-Directly Certified Students 



* 

- 5.8 



All Meals 



* 

- 3.3 




Paid Meals 



- 2.8 




Free/RP Meals 



-8 -6 -4 -2 0 2 4 6 8 10 

Percentage Point Impact 

* Estimated impact is statistically significant at the 0.05 level. 
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The negative impact of UFD on the overall participation rate could have different 
interpretations depending on which students were participating at lower levels as a result of the 
pilot intervention. The intervention was designed to reduce certification for free or reduced-price 
meals among students who were income-ineligible for benefits without reducing certification 
among income-eligible students. Correspondingly, it was designed to reduce free and reduced- 
price participation among income-ineligible students without altering free or reduced-price 
participation among students whose incomes made them eligible for these benefits. 

Based on impact estimates from the evaluation, UFD had the opposite effect. Among 
income-ineligible students, the estimated impact of UFD on the free/reduced-price NSLP 
participation rate was very small (-0.2 percentage points) and not statistically significant. 
Among income-eligible students, by contrast, UFD was estimated to lead to a statistically 
significant 9.3 percentage point decline in the free/reduced-price participation rate. Among both 
groups, the estimated impact on the paid participation rate was not statistically significant. 

These estimates of the impact of UFD on NSLP participation rates are consistent with the 



Estimated Impact of Up-Front Documentation on Rates of 
Free/Reduced-Price NSLP Participation, by Income Level 
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Percentage Point Impact 



FPL = Federal Poverty Level 

* Estimated impact is statistically significant at the 0.05 level. 



observed changes in the participation rate during the first year in which the UFD pilot was in 
place. While the evaluation suggested that UFD led to a decline in the overall free/reduced-price 
participation rate of about 15 percent, the observed free/reduced-price participation rate in the 
UFD pilot districts declined by 16 percent between the year before UFD and the first year it was 
in place. 
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The Impacts of Graduated Verification 

Overall, GV was estimated to lead to a small reduction in the participation rate of -3.4 
percentage points, which represented a 4 percent decline from the rate of 81.5 percent in 
comparison districts. This estimated impact was not statistically significant at the 0.05 level. 
The estimated impact on the free/reduced-price participation rate was larger in magnitude, at 
-4.7 percentage points, which represented a 14 percent decline from the rate of 33.5 percent in 
comparison districts, but this estimate was not statistically significant either. 



Estimated Impact of Graduated Verification on Rates of 
NSLP Participation Among Non-Directly Certified Students 




-6 -4 -2 0 2 4 6 8 10 

Percentage Point Impact 

None of the estimated impacts is statistically significant at the 0.05 level. 



The estimated impact of GV on the free/reduced-price participation rate was negative among 
both students from households with incomes of no more than 185 percent of the federal poverty 
level and among students from households with higher incomes. In each case, however, the 
estimated impact was not statistically significant. Among those from households with incomes 
of no more than 185 percent of poverty (and thus income eligible for free or reduced-price 
meals), GV was estimated to lead to a 6.8 percent decline in the free/reduced-price participation 
rate. Among higher-income households, the estimated impact was -2.6 percentage points. 
However, among the lowest- income households — those of no more than 130 percent of poverty 
and thus income-eligible for free meals — GV was estimated to lead to a 13.3 percent decline in 
the participation rate. This estimated impact was statistically significant at the 0.05 level. 



XI 



Estimated Impact of Graduated Verification on Rates of 
Free/Reduced-Price NSLP Participation, by Income Level 
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I. INTRODUCTION 



The National School Lunch Program (NSLP) and School Breakfast Program (SBP) serve 
nearly 4 billion free and reduced-price meals annually to children certified as being from low- 
income households (U.S. Department of Agriculture 2003). In recent years, however, 
policymakers have raised concerns about the integrity of the programs’ process for establishing 
eligibility for these benefits. In response, the U.S. Department of Agriculture (USDA) asked 
school districts around the country to voluntarily participate in the National School Lunch 
Program Application/Verification Pilot Projects to test ways of improving the process for 
certifying students for free and reduced-price meals. USDA also contracted with Mathematica 
Policy Research, Inc. (MPR) to conduct an evaluation of two of the approaches that were tested: 
(1) Up-Front Documentation (UFD), and (2) Graduated Verification (GV). 

This report presents the impacts of UFD and GV on the extent to which students participate 
in the NSLP by getting a meal for which the School Food Authority (SFA) seeks federal 
reimbursement. A companion study to this report (Burghardt et al. 2004) described the findings 
of the evaluation on the effects of UFD and GV on the certification of income-eligible and 
income-ineligible students for free or reduced-price benefits. The analysis presented in this 
report extends the analysis of certification to examine how the demonstration affected the 
participation of eligible and ineligible students, as well as the participation of certified and 
noncertified students. 

The rest of this chapter presents important background information for the study, lists the 
key research questions being addressed, and briefly summarizes the relevant prior research. 
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A. STUDY BACKGROUND 



Several studies examining income levels of students certified for free or reduced-price meals 
have found that a nontrivial number of these students have income levels that make them 
ineligible for the level of benefits they are receiving (U.S. Department of Agriculture 1990, and 
Office of Inspector General, USDA 1997). To address this issue, USDA sought school districts 
to test alternative ways of determining the income eligibility of students’ families. USDA 
published a report on first-year experience of these districts (U.S. Department of Agriculture 
2002) to describe changes in the districts that implemented UFD and GV, and it contracted for 
the evaluation to estimate the impacts of these pilot projects on selected outcomes. In particular, 
the evaluation included nine districts that tested UFD during the 2000-2001 through 2002-2003 
school years and three districts that tested GV during these same years. 

Under UFD, districts required that all applicants for free or reduced-price meals provide 
documentation of their income or food Stamp/Temporary Assistance for Needy Families (TANF) 
receipt with their application.^ If the application did not include documentation, the district was 
required to inform the household that the application was incomplete and request that the 
household resubmit the application with documentation. Without documentation, the district 
could not certify a student for free or reduced-price meal benefits. After the district reviewed 
and approved applications, it was not required to perform the verification of income for any free 
or reduced-price approved children, whereas, under federal regulations, a small sample of 
approved households typically is verified. Students certified for free meals through direct 
certification in the UFD pilot districts were not subject to these requirements, which applied only 
to students who submitted an application. 

* For additional details on the pilot projects and how their rules differed from standard 
district eligibility determination procedures, see Burghardt et al. (2004). 
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Under GV, application procedures were strengthened and, in certain circumstances, the 
verification process was enhanced. First, households that applied for free or reduced-price meals 
and whose benefits had been terminated or reduced in the prior year because of the district’s 
verification procedures had to provide documentation of their incomes or of their categorical 
eligibility at the point of application. Second, the district was required to conduct the standard 
verification of three percent of participating households and the following additional 
verifications: 



• If 25 percent or more of the originally verified applications led to a termination or 
reduction of free or reduced-price meal benefits, the district was required to verify an 
additional 50 percent of remaining applications. 

• If 25 percent or more of these second-round verifications resulted in terminations or 
reductions in benefits, the district was required to verify all remaining applications. 



B. RESEARCH QUESTIONS 

The evaluation examines the overall effects of UFD and GV on the NSLP. Burghardt et al. 
(2004) presented estimates of the impacts on certification for free or reduced-price meals among 
students who were income eligible and students who were not income eligible for the benefits. 
In this report, we present estimates of the impacts of UFD and GV on NSLP participation that 
address four sets of research questions: 



1. How do UFD and GV affect the percentages of students who get free meals, 
reduced-price meals, and paid meals on a typical school day? How do the pilots 
affect the overall NSLP participation rate? 

2. How do UFD and GV affect participation rates among students income eligible 
for benefits? How do the pilots affect participation rates among students not 
income eligible for free or reduced-price meals? 

3. How do UFD and GV affect the rate of participation accuracy — that is, the 
percentage of students getting free or reduced-price meals who are income 
eligible for these benefits? 
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4. What is the proportion of school lunch reimbursements paid in excess of the 
amount that should be paid based on students’ true income eligibility levels in 
UFD and GV pilot and comparison districts? 

C. PRIOR RESEARCH 

The findings of the evaluation with respect to the impacts of UFD and GV on application 
rates (Hulsey et al. 2004) and certification rates (Burghardt et al. 2004) for free and/or reduced- 
price meals provide some guidance as to what we may expect the impacts of the pilots on 
participation to be. Those reports found that the estimated impacts of UFD and GV on 
certification among students income ineligible for free or reduced price meal benefits were not 
statistically significant. In other words, the evaluation found no statistically significant evidence 
that the pilots deterred income-ineligible students from applying for, and being approved for, 
free or reduced-price meals. On the other hand, the pilots reduced application rates and 
certification rates among income eligible students. Thus, the pilots created barriers to 
certification among some students who qualified for free or reduced-price meal benefits. Finally, 
the estimated effects of UFD and GV on rates of certification accuracy — the percentage of 
students certified for free or reduced-price meals who were income eligible for the benefits they 
were receiving — were not statistically significant. 

Previous research has shown that certification and participation rates are positively related — 
students certified for free or reduced-price meals are more likely than otherwise similar 
noncertified students to get a school lunch (see, for example, Gleason 1995). If this relationship 
holds for the students in UFD and GV districts in the evaluation, in light of the findings in 
Burghardt et al. (2004) cited above, we expected the following impacts of the pilots on 
participation: 

• A decline in NSLP participation of income eligible students 
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• No significant change in the NSLP participation of income ineligible students 

• A decline in overall NSLP participation 

A U.S. Department of Agriculture report titled: NSLP Application/Verification Pilot 
Projects: Report on First Year Experience (2002) used aggregate administrative data to compare 
participation in the pilot districts during the first year of pilot implementation with participation 
during the period just before pilot implementation. Both UFD and GV districts had reductions in 
free and reduced-price meals and increases in paid meals served to students in the first year of 
implementation compared to the previous year. The net effects of these offsetting changes were 
modest reductions in total meals served after pilot implementation. One limitation of this 
analysis, however, was that it could not distinguish between free or reduced-price meal 
participation among students income eligible for these benefits versus free or reduced-price meal 
participation among income-ineligible students. 

Since Burghardt et al. (2004) found that the estimated effect of the pilots on the rate of 
certification accuracy was not statistically significant, we had no strong a priori expectations 
about whether the impact on participation accuracy would be positive or negative. Nor do we 
have a strong expectation about the direction or magnitude of the impact of the pilots on the rate 
of reimbursements paid in excess of the amount that should be paid based on students’ true 
eligibility levels. 
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II. DATA AND METHODS 



The methods we used to estimate the impacts of Up-Front Documentation (UFD) and 
Graduated Verification (GV) on participation levels and participation accuracy in this report 
build directly on the methods used to estimate impacts on certification levels and certification 
accuracy as described in Burghardt et al. (2004).' The study used a comparison group design, in 
which a single comparison district was selected to match each UFD and GV pilot district in the 
evaluation. We interviewed parents of samples of students selected from each pilot district and 
a matched comparison district and used survey and administrative data to measure their 
outcomes (certification status, certification eligibility, and participation). The difference 
between mean outcomes in pilot versus comparison districts, after controlling for other 
observable characteristics in a regression framework, was the estimate of the impact of UFD and 
GV on these outcomes. 

The student samples were selected at random from among all students in the pilot and 
comparison districts, except for those who had been directly certified to receive free meals on the 
basis of receipt of food stamps, TANF, or assistance from the Food Distribution Program on 
Indian Reservations (FDPIR). Thus, the results presented in this report generalize to all 



* For a detailed description of the methodology used to estimate the impacts of the pilots on 
certification and certification accuracy, see Volume II of Burghardt et al. (2004). 

2 

A drawback of a nonexperimental comparison group design such as this is that it is 
difficult to identify and successfully recruit comparison districts that are similar to pilot districts 
in all ways except for the pilot intervention. We developed a careful process for selecting 
comparison districts that we felt would meet this objective, and, for the most part, the resulting 
characteristics of students in comparison versus pilot districts were similar. Nonetheless, we 
could not identify or control for unobservable differences between the two groups of districts. 
See Burghardt et al. (2004) for a description of the comparison district selection process, as well 
as the observable characteristics of students in pilot versus comparison districts. 
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non-directly certified students in the evaluation districts. The reason for the exclusion of directly 
certified students was that the UFD and GV pilot interventions directly affected only those 
students who were not directly certified. For selected tables, however, the report also includes 
estimates of the rates of participation and participation accuracy for the full group of students, 
including those directly certified. These estimates are based on a set of assumptions made about 
the levels of participation and rates of participation accuracy using a methodology described in 
greater detail later in this chapter. 

The rest of this chapter describes details of the methods we used to address the research 
questions listed in Chapter I. Section A describes the data and methods we used to estimate 
NSLP participation. Section B lists the outcome measures we examined, summarizing how we 
used the NSLP participation data to create these outcome measures. Section C presents the 
methodology we used to estimate impacts of UFD and GV on these outcomes. Finally, Section 
D reports on some limitations of the study design. 

A. MEASURING NSLP PARTICIPATION 

The primary data source for measuring NSLP participation levels in the pilot and 
comparison districts in this report was the survey of the parents or guardians of sampled students 
in these districts. We contacted parents, by telephone and thru in-home interviews, between 
October 2002 and January 2003. We asked them, among other questions, how many days per 
week their children usually ate a school lunch. We then divided the reported number of meals in 
a usual week by five to estimate each student’s likelihood of getting a school lunch on any given 

We tried to minimize the influence of the exclusion of directly certified students on the 
impact estimates from the evaluation by selecting comparison districts that matched the pilot 
districts in their use of direct certification. For further discussion of this issue, see Burghardt et 
al. (2004). 
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day. By averaging this likelihood of participating on a given day across all sampled students in a 
district, we obtained an estimate of the overall daily participation rate of students in that district, 
excluding those who had been directly certified. 

To estimate participation rates among all students, including those directly certified, we 
relied on the assumption that patterns of participation among directly certified students would be 
similar to those among non-directly certified students with incomes of no more than 130 percent 
of poverty in the appropriate comparison district. In other words, for directly certified students 
in a given pilot district, the participation rate is assumed to be equal to the participation rate 
among non-directly certified students who are income eligible for free meals in the comparison 
district to which that pilot district is matched. For directly certified students from that 
comparison district, the participation rate assumption is the same."^ This assumption, along with 
information on the percentage of all students who were directly certified, allowed us to estimate 
the overall participation rate. 

These estimates were based on reports by each student’s parent/guardian about whether the 
student usually got a school lunch. Because of the burden involved, the study did not gather 
individual-level administrative data on whether each sampled student received a reimbursable 
school meal over a specific period. 

The other key piece of information used in measuring NSLP participation was information 
obtained from school records on the certification status of each student in the sample.^ This 
information enabled us to classify each student selected for the study as certified for free meals, 

^ The reason that the same participation rate assumption is made for matched pairs of pilot 
and comparison districts is that we wanted to ensure that we did not create an impact on 
participation among this group by assumption. 

^ See Burghardt et al. (2004) for a description of the collection of school records data on 
students’ certification status. 
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certified for reduced-price meals, or not certified. Correspondingly, we used certification status 
to classify the meals the student obtained as either free, reduced-price, or paid. 

An important limitation of the study’s participation data is that parents reported the data. 
This is a limitation because parents may not know whether their children actually got school 
lunches. For example, parents may not know the extent to which their children obtain NSLP 
reimbursable lunches when they are at school. Previous research indicates that parents typically 
overestimate their children’s true levels of participation as measured by administrative counts of 
reimbursable meals claimed by SFAs. Burghardt et al. (1993) and Gleason and Suitor (2001), 
for example, found substantially higher rates of participation when estimates were based on 
parental reports of usual participation than when based on foods consumed from the school 
cafeteria on a given day. 

Due to this potential misreporting of students’ NSLP participation by their parents, we 
sought an alternative way of both measuring NSLP participation in the pilot and comparison 
districts and estimating the impact of the pilot on participation. An alternative method of 
measuring NSLP participation rates in pilot and comparison districts is to use SFA 
administrative records to provide aggregate counts of reimbursable meals claimed by students. 
FNS regularly collected NSLP administrative data from UFD and GV pilot and comparison 
districts during the years of pilot operation and in two prepilot years. Data FNS collected 
included the number of free, reduced-price, and paid meals served in the district by month, along 
with the number of serving days and the number of students enrolled in the district. This 
information allowed us to calculate participation rates, defined as the percentage of all students 
in the district who got a school lunch (overall, or by meal price status — free, reduced-price, or 
paid) on a given day. This methodology provides an accurate estimate of overall participation at 
the district level, but it cannot be used to determine the participation status of individual students 
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in the district (or in our sample). Without individual-level participation data, we would not be 
able to estimate participation rates separately for students who were income eligible versus those 
who were income ineligible. Nor could we use the other individual-level information collected 
in the evaluation surveys to control for factors unrelated to the pilot intervention that may have 
influenced participation. 

Because the parent-reported data on participation from the survey were available at the 
student level, we use that source of information throughout this report. Because of the 
possibility that a parent-reported measure of NSLP participation is not accurate, however, we 
also used the administrative data on participation to conduct a sensitivity analysis to assess 
whether our results would likely have been different if we had more accurate individual-level 
participation data. In general, the sensitivity analysis corroborated the findings reported in 
Chapter III. Appendix A describes this sensitivity analysis and its results. 

B. OUTCOME MEASURES 

We used the data on NSLP participation rates to define three main types of outcome 
variables for our analysis: (1) participation rates, (2) rates of participation accuracy, and (3) rates 
of reimbursement accuracy. We were interested in measuring these outcomes for all students 
and for selected subgroups of students as defined by their income eligibility and certification 
statuses. Table II. 1 shows one way of conceptualizing the relevant categories of all students and 
of NSLP participants. 

The top panel of Table II. 1 is a modified version of Table II. 1 in Burghardt et al. (2004). 
This panel shows how the total population of students in a district is distributed across different 
categories of certification status and income eligibility status. For example, the group Ne 
represents the number of students certified for free meals by application in the district (regardless 
of income eligibility status). The bottom panel has a similar structure, but instead of showing 
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TABLE II. 1 



POSSIBLE COMBINATIONS OE INCOME AND CERTIEICATION STATUS, 
ALL STUDENTS AND NSLP PARTICIPANTS 



Income Eligibility Status 





<= 130 Percent 131 to 185 Percent 






Certification Status 


PPL“ 


PPL 


>185 Percent PPL 


Total 


All Students 


Directly Certified’’ 


Na 


— 


— 


Na 


Certified Pree by Application 


Nb 


Nc 


Nd 


Np 


Certified Reduced-Price 


Np 


Ng 


Nh 


N, 


Not Certified (Paid) 


Nj 


Nk 


Nl 


Nm 


Total 


Nn 


No 


Np 


No 


NSLP Participants 


Directly Certified’’ 


Pa 


— 


— 


Pa 


Certified Pree by Application 


Pb 


Pc 


Pd 


Pp 


Certified Reduced-Price 


Pp 


Pg 


Ph 


Pi 


Not Certified (Paid) 


Pj 


Pk 


Pl 


Pm 


Total 


Pn 


Po 


Pp 


Po 



“This column potentially includes both students receiving food stamp or Temporary Assistance for Needy Eamilies 
(ES/TANE) benefits and students not receiving these benefits. Eor simplicity, we assume that no ES/TANE 
recipients have incomes above 130 percent of the EPL. 

’’We assume that all directly certified students have incomes below 130 percent of the EPL. 

“If a student obtained an NSLP lunch, the student is assumed to be a participant in the meal price status category for 
which he or she is certified, or if the student is not certified, he or she is assumed to be a paid participant. Our 
analysis assumes meal counting and claiming for reimbursement purposes were performed correctly. 

PPL = federal poverty level. 
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how all students are distributed across these groups, it shows how school lunch participants are 
distributed across the groups. For example, Pe represents the number of students who are 
certified for free meals by application who got a school lunch on a given day. 

Together, the two panels of Table II. I can be used to define the various outcome measures 
examined in this report. Table II.2 lists these outcomes and defines them in terms of the groups 
shown in Table II. 1. We describe the outcomes in greater detail below. 

1. Participation Rates 

As described earlier, the NSLP participation rate is defined as the likelihood that the student 
ate a school lunch on any given day and is based on a parental report on the household survey. 
We calculated the participation rate by dividing the reported number of meals the student usually 
ate per week by five. By calculating the mean of this variable across all students in our sample 
in a given district, we estimated the NSLP participation rate among non-directly certified 
students in that district. For calculating the percentage of all students who ate free, reduced- 
price, or paid meals, we used information on the certification status of each student that we 
obtained from the district to assign the meals that students received to the free, reduced-price, or 
paid category. 

We derived the other participation rate outcomes by limiting the group for which the 
participation rate was calculated. In particular, we measured the rate of participation among 
students income eligible for free (or free/reduced-price) meals by restricting the sample to 
students with estimated incomes of no more than 130 percent (185 percent) of the federal 
poverty level (FPL). Similarly, we measured the rate of participation among income ineligible 
students by restricting the sample to students with incomes above 185 percent of the FPL. 
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TABLE II.2 



OUTCOME MEASURES USED IN ESTIMATING IMPACTS ON PARTICIPATION 



Outcome Measure 


Population/Sample" 


Participation Rates 




PI: Probability that student obtained a school lunch on typical day 


o 
' — . 

o 


P2: Probability that student obtained a free school lunch on typical day 


Pe/Nq 


P3: Probability that student obtained a reduced-price school lunch on 
typical day 


Pi/Nq 


P4: Probability that student obtained a paid school lunch on typical day 


Pm/Nq 


P5: Probability that student income eligible for free meals obtained a 
school lunch on a typical day 


Pn/Nn 


P6: Probability that student income eligible for free meals obtained a 
free school lunch on a typical day 


Pb/Nn 


P7: Probability that student income eligible for free meals obtained a 
reduced-price school lunch on a typical day 


Pf/Nn 


P8: Probability that student income eligible for free meals obtained a 
paid school lunch on a typical day 


Pj/Nn 


P9: Probability that student income eligible for free or reduced-price 
meals obtained a school lunch on a typical day 


(Pn + Po) / (Nn + No) 


PIO: Probability that student income eligible for free or reduced-price 
meals obtained a free school lunch on a typical day 


(Pb + Pc)/(Nk + No) 


PI 1: Probability that student income eligible for free or reduced-price 
meals obtained a reduced-price school lunch on a typical day 


(Pf + Pg)/(Nn + No) 


PI 2: Probability that student income eligible for free or reduced-price 
meals obtained a paid school lunch on a typical day 


(Pj + Pk) / (Nn + No) 


PI 3: Probability that student not income eligible for free or reduced- 
price meals obtained a school lunch on a typical day 


Pp/Np 


P14: Probability that student not income eligible for free or reduced- 
price meals obtained a free school lunch on a typical day 


Pd/Np 


P15: Probability that student not income eligible for free or reduced- 
price meals obtained a reduced-price school lunch on a typical day 


Ph/Np 


P16: Probability that student not income eligible for free or reduced- 
price meals obtained a paid school lunch on a typical day 


Pl/Np 
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TABLE II. 2 {continued) 



Outcome Measure 


Population/Sample" 


Rates of Participation by Certification Status 




P_CF: Probability that student certified for free meals obtained a 
school lunch on a typical day 


Pe/Ne 


P_CRP; Probability that student certified for reduced-price meals 
obtained a school lunch on a typical day 


Pi/Ni 


P_NC: Probability that student not certified for free or reduced-price 
meals obtained a school lunch on a typical day 


Pm / Nm 


Rates of Participation Accuracy 




Free Meal Accuracy 




PA_F: Percentage of free meals served to students income eligible 
for free meals 


Pb/Pe 


Free/Reduced-Price Meal Accuracy 




PA_FRP1: Percentage of free or reduced-price meals served to 
students income eligible for the level of benefits they received 


(Pb + Pg)/(Pe + Pi) 


PA_FRP2: Percentage of free or reduced-price meals served to 
students not overcertified for the level of benefits they received 


(Pb + Pg+Pf)/(Pe + Pi) 


PA_FRP3: Percentage of free or reduced-price meals served to 
students with incomes below 185 percent of the FPL 


(Pb + Pg+Pf + Pc)/(Pe + Pi) 


Rates of Reimbursement Accuracy 




RA_OP: Percentage of all federal reimbursements for meals served to 
students not eligible for the level of benefits they received 


[0.40Pc+ L94 Pd-h 1.54Ph]/ 
(2.29Pe-h 1.89Pi-f0.35Pm) 


RA_UP: Percentage shortfall in federal reimbursements because some 
certified students received a lower benefit than they were eligible for 


0.40Pp / 

[2.29Pe+ 1.89Pi-f0.35Pm ] 



“The formulas shown here are all designed to show rates of participation, participation accuracy, and certification 
accuracy among non-directly certified students, since that is the group that forms the basis for most of the analysis 
in the report. Thus, where relevant, the formulas should be assumed to exclude the groups and P^. 

FPL = federal poverty level. 
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In defining these subgroups, we used income information collected in both telephone and in- 
person surveys as part of the evaluation. We developed a process to try to minimize the extent of 
reporting error in the income data and classify students into income categories as accurately as 
possible. We also conducted a sensitivity analysis to determine whether the estimated impacts of 
the pilot on the rate of certification changed under different ways of defining income. We found 
that the results were robust to changes in the specifics of the process for defining income 
eligibility categories. See Burghardt et al. (2004) for more information about the process used to 
define income eligibility categories and the sensitivity of the results to the income eligibility 
definition used. 

As mentioned previously, we estimated the participation rate among all students in the 
district (including those directly certified), by using information on the percentage of students in 
the district who were directly certified and relying on the assumption that directly certified 
students participate at the same rate as non-directly certified students in the same income 
category (that is, with incomes of no more than 130 percent of poverty). To calculate the 
participation rate among all students, for example, we used the following formula: 

(1) Pr(M) = Pr( M I NDC )*{1 - Pr(DC) j + Pr( M \ DC )* Pr( DC ) 

= Pr ( M I NDC )*{ 1 - Pr(DC) } -i- Pr( M I NDC, Income<=130,Comp.Dist.)* Pr( DC ) 

In this equation, the participation rate among all students is represented by the probability of 
getting a school meal (M). This is equal to the weighted sum of two terms. The first term is the 
participation rate among non-directly certified students, or the probability of getting a meal 
conditional on no direct certification (NDC), and the weight on this term is the probability of not 
being directly certified (1 - Pr(DCJ). The second term is the participation rate among directly 
certified students, which — given the above assumption — can be written as the probability of 
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getting a meal conditional on not being directly certified, having a household income of no more 
than 130 percent of poverty, and being in the appropriate comparison district. The weight on this 
second term is the probability of being directly certified (Pr(DCj). 

2. Rates of Participation Accuracy 

Participation accuracy rates measure the percentage of free or reduced-price meals served to 
students eligible for the level of benefits for which they are certified. For example, free meal 
participation accuracy is defined as the percentage of all free meals consumed by students whose 
household incomes are less than 130 percent of the FPL or who receive food stamps or TANF. 
Conversely, 100 minus the participation accuracy rate is the measure of the percentage of meals 
served to students who were ineligible to receive them. 

These measures of participation accuracy correspond to the measures of certification 
accuracy presented in Burghardt et al. (2004). That report presented four certification accuracy 
measures — one measure of free meal certification accuracy and three measures of free/reduced- 
price certification accuracy. Each of these four measures was calculated separately for a 
population that excluded directly certified students and for a population that included directly 
certified students. Analogously, we present two sets of four participation accuracy measures in 
this report — for both non-directly certified students and all students, in other words, we present 
the measure of free meal participation accuracy described in the preceding paragraph and three 
measures of free/reduced-price participation accuracy. In each case, as Table II.2 shows, the 
participation accuracy measure focuses on a group of participants (thus excluding students who 
do not get school meals) to determine whether the level of benefits they are receiving (free, 
reduced-price, or paid) is consistent with their household circumstances. 

The four definitions of free/reduced-price participation accuracy differ in how they define 
free/reduced-price participants as being eligible, or accurate. The first measure treats 
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free/reduced-price participants as being accurate if their household circumstances make them 
income eligible for exactly the level of benefits they are receiving — that is, if their income is less 
than 130 percent of the FPL and they are free meal participants or if their income is between 130 
and 185 percent of the FPL and they are reduced-price meal participants. The second measure of 
free/reduced-price meal accuracy loosens the definition of accuracy somewhat, treating 
participants as accurate as long as they are not “overcertified” — that is, certified for a higher 
level of benefits than that for which they were eligible. Thus, free meal participants with 
incomes of less than 130 percent of poverty and reduced-price participants with incomes of less 
than 185 percent of poverty would be considered accurate, while all other free/reduced-price 
meal participants would be considered inaccurate. Therefore, meals received by students 
certified for reduced-price meals but eligible for free meals would be counted as inaccurate in the 
first measure but accurate in the second one. The third measure is the most inclusive measure of 
free/reduced-price meal participation accuracy — any free or reduced-price meal participant with 
an income less than 185 percent of poverty is considered accurate. In this measure, meals 
received by students certified for free meals but eligible for reduced-price meals are counted as 
accurate, whereas in the first and second measure such meals are counted as inaccurate. 

3. Reimbursement Accuracy 

The final outcome measure is similar to the participation accuracy measure but takes into 
account that different types of inaccuracy have different financial consequences. These 
differential consequences arise from the fact that USDA reimburses NSLP lunches at three levels 
for free, reduced-price, and paid lunches. In particular, lunch reimbursements are specified in 
the National School Lunch Act. For the 2002-2003 school year. Section 4 of the Act specified a 
reimbursement rate for all lunches served, and Section 1 1 specified additional payments for free 
and reduced-price meals. In addition, SFAs receive donated commodities for each meal served. 
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The overall per-lunch reimbursement rates in 2002-2003 were $2.29 for free lunches, $1.89 for 
reduced-price lunches, and $0.35 for paid lunches. These totals represent the sum of the Section 
4 and Section 1 1 funds, along with the value of donated commodities, as shown in Table II.3. 

Thus, a district that served a free school lunch in 2002-2003 to a student who was income 
eligible for only a reduced-price lunch was reimbursed $0.40 more than it should have been 
reimbursed according to the student’s income eligibility level. If, on the other hand, that student 
had been income eligible for neither free nor reduced-price meals, the district would have been 
reimbursed $1.94 more than it should have been. 

To measure reimbursement accuracy, we developed two measures of the prevalence of 
erroneous payments. The first is the overpayment rate, defined as the percentage of actual 
payments made to districts in excess of what the district would have received if no school lunch 
participants had been overcertified. The second is the rate of underpayments among certified 
students. This is defined as the amount of additional payments that should have been — but were 
not — made to districts due to participants being income eligible for free meals but certified for 
reduced-price meals only, measured as a percentage of actual payments made to the district.*’ 

We took the following steps to measure these two rates of erroneous payments. We first 
calculated average reimbursement amounts under three alternative scenarios within each of the 
districts in the evaluation: 

• Scenario 1: Actual Reimbursements. Reimbursements under Scenario 1 were 
defined as the reimbursements actually paid per student in the district. For a given 
student, this was calculated as his or her probability of getting a school lunch on a 



^ This rate of underpayments among certified students does not take into account 
“underpayments” brought about by paid meal participants who were eligible for free or reduced- 
price meals. 
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TABLE II.3 



REIMBURSEMENT RATES PER MEAE EOR NSEP PUNCHES SERVED^ 



Type of Punch 


Section 4 


Section 11 


Value of 
Donated 
Commodities 


Total 


Pree 


0.20 


1.94 


0.15 


$2.29 


Reduced-Price 


0.20 


1.54 


0.15 


$1.89 


Paid 


0.20 


0.00 


0.15 


$0.35 



‘‘These are the reimbursement rates for the majority of SPAs. A few SPAs receive higher 
reimbursement rates, either because they have a large proportion of students eligible for free or 
reduced-price meals or because they are located outside of the continental United States. 
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typical day times the amount the school would have been reimbursed if that student 
did get a school lunch, which was determined by the student’s actual certification 
status ($2.29 for those certified for free meals, $1.89 for those certified for reduced- 
price meals, and $0.35 for noncertified students). For a given student, the equation 
below shows how the actual reimbursements were calculated: 

(2) R_actual = Pr(M)* [ 2.29*CF + 1.89*CRP + 0.35*(1-CF - CRP )] 
where M = gets a school meal 

CF = binary indicator of whether student is certified free 

CRP = binary indicator of whether student is certified reduced-price 



• Scenario 2: Reimbursements with No Overcertification. Under Scenario 2, we 
calculated the reimbursements per student the district would have received if no 
student were reimbursed for a benefit in excess of the one for which they were 
actually income eligible. This amount was calculated as the student’s probability of 
getting a school lunch times the maximum reimbursement for which the student was 
both income eligible and had been certified. If, for example, the student was 
certified for free meals but income eligible only for reduced-price meals, the level of 
reimbursement used in this calculation would have been $1.89 (the reduced-price 

•y 

reimbursement) based on their actual income eligibility. The equation below shows 
the calculation of reimbursements with no overcertification for a given student: 



(3) R_no_overcert = Pr(M)* [ 2.29*CF*FE + L89*(CF*RPE + CRP*FRPE) + 

0.35*(1-CFRP*FRPE )] 

where M, CF, and CRP are defined as before, and 

CFRP = binary indicator of whether student is certified free or 
reduced-price 

EE = binary indicator of whether student is income eligible for free 
meals 

RPE = binary indicator of whether student is income eligible for 
reduced-price meals 



The reimbursements calculated under this scenario did not take into account “under 
certification,” however. In other words, among students who were eligible for greater 
reimbursements than their current certification level. Scenario 2 reimbursements would include 
only the level of benefits they were receiving, not the higher level of benefits for which they 
were eligible. 
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FRPE = binary indicator of whether student is income eligible for free 
or reduced-price meals 

• Scenario 3: Reimbursements with Accurate Certification. Under Scenario 3, we 
calculated the reimbursements per student the district would have received if all 
certified students had been certified accurately according to their income eligibility 
level. This amount was calculated as the student’s probability of getting a school 
lunch times the reimbursement the district would have received if all certified 
students income eligible for free meals had been certified for free meals and if all 
certified students income eligible for reduced-price meals had been certified for 
reduced-price meals. Among noncertified students, the reimbursement level under 
this scenario was $0.35, regardless of the income eligibility status of the student. 
The following equation shows the calculation of reimbursements with accurate 
certification among participants for a given student: 

(4) R_accurate=Pr(M)*[2.29*FE*CFRP+1.89*RPE*CFRP+0.35*(l-CFRP*FRPE)] 
where M, FE, RPE, CFRP, and FRPE are defined as before. 



Next, we calculated the amount of overpayments to districts as the difference between 
reimbursements under Scenario 2 (that is, reimbursements with no overcertification) and 
reimbursements under Scenario 1 (that is, actual reimbursements). This difference can be 
interpreted as the amount per student (on a typical day) that a given district received in excess of 
what it would have received if no students had been overcertified. By dividing this difference by 
the actual reimbursements paid to districts, we generated an “overpayment rate,” or the amount 
of overpayments in a district as a percentage of the total amount of reimbursements. 

Similarly, we calculated the amount of underpayments to certified students as the difference 
between the amount of reimbursements with accurate certification (Scenario 3) and the amount 
of reimbursements with no overcertification (Scenario 2). This difference captures 
underpayments only among students who were certified for reduced-price meals when they were 
actually eligible for free meals. Note that it does not include underpayments among noncertified 
students who were income eligible for either free or reduced-price meals. By dividing the 
difference by the actual reimbursements paid to districts, we generated an “underpayment rate,” 
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or the amount of underpayments in a district as a percentage of the total amount of 
reimbursements . ^ 

C. APPROACHES TO MEASURING IMPACTS 

Throughout the impact analysis presented in Chapter III, we use survey data on participation 
in calculating the impact of the pilots on the outcome measures described above. We then 
examine the sensitivity of the results to the use of this survey data (versus the more accurate, yet 
less detailed, administrative data) in Appendix A. This section describes the methodology used 
to estimate impacts on participation rates and on participation accuracy. 

I. Impacts on Participation Rates 

At its most basic level, the estimated impact of the pilots on participation is the difference 
between the mean value of the participation outcome measure in the pilot districts and the mean 
value of the participation outcome measure in the comparison districts, controlling for other 
differences between the two types of districts. In Chapter III, we first show the simple mean 



o 

In analyzing reimbursement accuracy, we calculated mean overpayment and underpayment 
rates in pilot and comparison districts. We did not, however, estimate the impacts of the pilots 
on these rates, due to the complexity of developing and estimating these impact models (and 
limited project resources for doing so). In particular, the reimbursement accuracy outcome 
measures actually depend on three factors: (1) the frequency with which a given student 
participates, (2) the student’s certification status, and (3) the student’s income eligibility level. 
Although the pilot interventions could conceivably have influenced participation and 
certification, they could not have affected students’ income eligibility level. Thus, estimating the 
impact of the pilots on these outcomes would have required us to use indirect methods, much 
like the indirect methods used to estimate impacts on participation accuracy. The methods 
needed to estimate impacts on these reimbursement accuracy outcomes are even more complex 
than the participation accuracy models, however, since the models had to take into account an 
additional number of possible combinations of certification status and eligibility status. For 
example, while students not certified for free or reduced-price meals could be excluded from the 
participation accuracy models, they would need to be accounted for in reimbursement accuracy 
models, since districts are reimbursed to some extent for paid meals served to noncertified 
students. 
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values of outcomes measures in pilot and comparison districts. These descriptive results show 
the overall difference in the outcome measure in pilot and comparison districts, which results 
from some combination of the impact of UFD or GV and the effect of other differences in the 
characteristics of pilot and comparison districts. We used regression analysis to control for these 
other differences in pilot and comparison district characteristics to generate an estimate of the 
impact of the pilots. 

In particular, we estimated a reduced-form model, based on the following two-equation 
model of the relationships between the pilot demonstrations, certification for free/reduced-price 
meals, and NSLP participation: 

(5) C,. = Cl + ^ a,, {DP.^ *Pi] + e, 

j=2 k=\ 

(6) M. = C 2 + X^p, + X^p, + b,C, + ^ d^jDPy {DP., */’] + «! 

j=2 k=\ 

where C, represents the student’s certification status (free or free/reduced-price); Mi is a variable 
indicating whether a student participates in the NSLP; Xu, X2i, and Xst, represent student/ 
household characteristics; DPij is a binary indicator of whether student i attended the jth pilot- 
comparison district pair (where there are a total of K pilot comparison pairs); P, is a binary 
indicator of whether student i attended a pilot district; and Cy and Ui are random error terms. 

This two-equation model suggests that the pilot may affect NSLP participation in one of two 
ways. It may have an indirect effect, by first influencing whether or not a student becomes 
certified, which in turn influences the student’s NSLP participation. In the model, the effect of 
the pilot on certification is represented by the vector of coefficients, ajk, and the effect of 
certification on participation is represented by bs. The model also allows the pilot to influence 
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participation directly, independent of its effect on certification, represented by the vector of 
coefficients U 2 h 

By substituting the right-hand side of equation (5) for certification status in equation (6), we 
obtained a reduced-form version of this model: 



M,. - Cj + -I- 



Cl + X^.b, + X^p^ + £ d,pP,. +£ a,, {DP., *P,] + e. 



j=2 k=l 



+ E +E vDPik * ^ ] + M, 

>2 



k=\ 



(7) 



A. 

— [c2 + b^Cy ] -I- Xji -I- b^b^ ] -I- ^2i^2 ^3i^4 E .^27 ^5^1 j ] 



>2 



+E [^2k + \DPik */’] + [«,+ he, ] . 



k=\ 



We estimated this reduced-form model to determine the impacts of UFD and GV on NSLP 
participation. In particular, we regressed our measure of NSLP participation on a constant term, 
the household characteristics included in Xu and Xsi, the binary district pair variables, and the 
interactions between district pair and pilot status. In estimating this reduced-form model, we 
could not identify all of the separate parameters of the two-equation model. For example, we 
estimated the overall effects of the district-pair-pilot status interaction terms (a 2 k + bsajk), which 
reflect both the direct and indirect effects of the pilot on NSLP participation. We could not 
separately identify the direct and indirect effects.^ To determine the overall impact of the pilot 



^ To identify all the parameters of the two-equation model, we would have had to estimate a 
model such as a two-stage least squares model, which would have required one or more variables 
that strongly affect certification status but do not directly affect participation (that is, the 
variables included in X 2 i). Because of the difficulty in finding such “identifying variables” and 
because the most important objective of the participation analysis reported here was to identify 
the overall effect of the pilot on NSLP participation, we did not attempt to estimate a two-stage 
least squares model. 
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on NSLP participation, we estimated each of the district-pair-pilot status interaction coefficients, 
which represent the estimated impact of the pilot in each site, and then calculated a simple 
average of these impacts across all UFD sites or all GV sites. 

The main form of the dependent variable was a variable indicating the likelihood that a 
student ate a school lunch on a typical day, as reported by parents. As described earlier, this 
dependent variable was calculated by dividing a variable indicating the number of times the 
student usually ate a school lunch per week by five. In addition to the model based on this 
dependent variable, we also estimated separate models with the following different versions of 
the dependent variable: (1) a variable indicating the likelihood that the student ate a free school 
lunch on a typical day, (2) a variable indicating the likelihood that the student ate a free or 
reduced-price school lunch on a typical day, and (3) a variable indicating the likelihood that the 
student ate a paid school lunch on a typical day. These four versions of the model were 
estimated using samples representative of all district students (except those directly certified). 
We also estimated separate versions of the models using subgroups of students, including (1) 
students with incomes of no more than 130 percent of the FPL, (2) students with incomes of no 
more than 185 percent of the FPL, and (3) students with incomes above 185 percent of the FPL. 

2. Impacts on Participation Accuracy 

Participation accuracy is measured as the percentage of free or reduced-price meals served 
to students who are eligible for those benefits. As with certification accuracy, the impact of the 
pilots on participation accuracy cannot be estimated using a straightforward regression model 
because of technical issues concerning the effects of the explanatory variables — including pilot 
status — on the composition of the estimation sample. In particular, when we estimated the 
impact of the pilots on participation, we simply regressed the explanatory variables on an 
outcome variable indicating the student’s likelihood of participating on a given day. If we were 
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to use an analogous strategy here, we would first have to restrict the sample to certified students 
and then regress the student’s income eligibility status (that is, a binary variable indicating 
whether the student’s income is no more than 130 or 185 percent of the FPL) on pilot status and 
the other explanatory variables. The problem with a regression model such as this, however, is 
that the sample in the model is endogenous in that pilot status (and other student characteristics) 
would affect the sample upon which the regression was based rather than affecting the dependent 
variable in the model. 

To estimate impacts on participation accuracy, therefore, we used a procedure analogous to 
the procedure to estimate impacts on certification accuracy, described in Burghardt et al. (2004). 
For the impact of the pilot on participation accuracy among students who eat free school lunches, 
for example, we used the results from the regression models of participation among those 
eligible for free meals and not eligible for free meals. This estimated impact can be written: 

(8) Imp { Accuracy I FM, = 1 } = Pr {FE, = 1 1 A, , FM,. = 1, = 1}- Pr {FF,. = 1 1 A,. , FM,. = 1, = 0}, 

where: FM; = indicator of whether student i eats a free school lunch on given day 

Xi = vector of relevant characteristics of student i 

FEi = binary indicator of whether student i is eligible for free meals 

Although we cannot directly estimate the impact of the pilot on the probability of being 
eligible for free meals, we can estimate the reverse — the impact on the probability of eating a 
free school lunch conditional on the student being income-eligible. Hence, we can estimate the 
conditional probability of eating a free lunch given being eligible for free meals and being in a 
pilot district (or comparison district). We can use Bayes’ Theorem, along with information on 
the proportion of the sample who are eligible and not eligible for free meals, to generate the 
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conditional probabilities we need to generate the estimated impact on accuracy. We did this as 



follows: 



Pr{F£. = 1|X;,FM, = 1,P = l} = 



Pr{FE_ =1&FM_ =l\X.,P = i} 
Pr{FM, =l|X; =l,P = l} 



(9) 



Pr{FM, = l| X;,FE =l,Pi= l}Pr{EE =11/’=!} 
Pr{FM, =l\X.,P=l} 



Each of the probabilities on the right-hand side of this equation is known. The conditional 
probabilities of eating a free school lunch were estimated previously, and the probability of being 
eligible for free meals is being treated as exogenous and was calculated from the sample. 

To estimate impacts on participation-based free meal accuracy, we calculated the value of 
the left-hand side probability in equation (9) separately for each of the pilot districts and then 
averaged these estimates. Next, we estimated an analogous set of conditional probabilities of 
free meal income eligibility given free meal participation for the comparison districts and 
averaged those estimates. The difference between the estimated probability of being income 
eligible for free meals conditional on having eaten a free meal in pilot districts and the 
corresponding conditional probability in comparison districts is the estimated impact on free 
meal participation accuracy. The impacts of UFD and GV on free or reduced-price meal 
accuracy can be estimated analogously. As with the certification-based accuracy measure in 
Burghardt et al. (2004), we estimated the standard errors of these impact estimates using a 
bootstrapping method. 

D. LIMITATIONS OF STUDY DESIGN 

Like all studies, this one is subject to several limitations that should be clearly understood so 
that the findings are used appropriately. We note these here. 
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1. Comparison Group Design 

The impacts of the pilot projects on deterrence, barriers, and accuracy are measured by 
comparing the outcomes of families in the pilot districts with the outcomes of families in 
comparison districts. The identification of suitably matched comparison districts was done with 
great care, and we controlled statistically for a range of personal and family characteristics that 
could influence the outcomes. As shown in Burghardt et al. (2004), the comparison district 
matching process produced a set of comparison districts and a sample of families with 
characteristics similar to those of the pilot districts. However, while — in our judgment — these 
comparison districts provide a reasonable basis for measuring net impacts of the pilot 
interventions, there remains an element of uncertainty about the quality of the benchmark that we 
cannot quantify. 

2. Sample Size Limitation 

As in most studies, resource constraints limited the size of the samples that it was possible to 
interview. For most outcomes, the study samples are of sufficient size to give us confidence that 
if the pilot projects caused an impact of a policy-relevant magnitude — such as an impact of 20 to 
30 percent of the mean outcome in comparison districts — our sample would provide a high 
likelihood of detecting the impact. For some measures, however, limits on sample sizes 
constrain our ability to detect significant demonstration effects. For example, in examining 
impacts on free/reduced-price participation among students not eligible for free/reduced-price 
certification, the sample cannot reliably detect impacts of a small absolute magnitude, even 
though these impacts are fairly large in relative terms. This is because we found the level of the 
outcome measure in comparison districts is a small absolute number. Suppose, for example, that 
the mean free/reduced-price participation rate among students not eligible for free/reduced-price 
certification in comparison districts were four percent. If that were the case, given our sample 
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sizes, we would be unlikely to be able to detect an impact of 1 percentage point, even though this 
impact would represent a 25 percent decline in the participation rate among this group. 

3. Issues of Generalizability 

The small number of demonstration sites and the voluntary nature of the decision to 
participate in the pilot projects necessarily limit our ability to draw conclusions about what 
would happen if the policies tested were to be implemented in a larger set of districts or 
nationwide. Just nine districts included in the study implemented UFD, and just three 
successfully implemented GV. Furthermore, these districts were part of a very small group 
nationwide that volunteered to test new procedures designed to improve the accuracy of the 
process for administering NSLP certification. Burghardt et al. (2004) document how these 
districts as a group compare with the nation as a whole in some readily observable characteristics 
of the districts. However, one can only speculate about how these districts differed from others 
nationwide in unobservable characteristics that may to affect the outcomes of interest in the 
evaluation. 
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III. IMPACTS OF THE PILOT PROJECTS ON PARTICIPATION 



This chapter presents estimates of the impacts of Up-Front Documentation (UFD) and 
Graduated Verification (GV) on participation in the NSLP in two parallel sections. Each section 
examines how the pilot proeedures ehanged the likelihood that a student got a reimbursable 
school lunch and the likelihood that he or she got a free, reduced-price, or paid sehool lunch. We 
present separate impaet estimates for all students and for groups of students defined by their 
eligibility status. To provide context for the impact estimates, we present mean values of each of 
the outcome measures for both pilot and comparison districts. Finally, each section examines 
participation accuracy and reimbursement accuracy. 

Burghardt et al. (2004) presented estimates of the impaets of the pilots on levels of 
certification among students in UFD and GV districts, and these estimates help us form 
expeetations about what we might find in examining the pilots’ effects on participation. As 
described in Chapter I, that study found that UFD and GV reduced certification, primarily among 
income-eligible (low-ineome) students. Since lower levels of certifieation imply that fewer 
students can potentially get free or reduced-price meals, we would expect the pilots to reduce 
free and redueed-price participation. Furthermore, since certified students are more likely to 
participate than noncertified students of the same income level (Gleason 1995), the pilots may 
also have redueed overall participation. However, whether and to what extent overall 
participation is reduced also depends on whether those students who were discouraged by the 
pilots from becoming certified deeided to get a sehool luneh at the full priee. Indeed, evidenee 
from FNS analysis of the first-year pilot experience indicates that free and reduced-price 
partieipation deelined after pilot implementation, while paid participation increased. (U.S. 
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Department of Agriculture 2002)^ Finally, it is possible that UFD or GV pilots had a spillover 
effect, whereby students whose certification status was not affected by the pilot procedures — 
primarily those from families with incomes exceeding 185 percent of the FPL — changed their 
participation behavior because of the pilot procedures. For example, the spillover effect could 
take the form of a peer effect whereby students who were not directly affected by the pilot could 
have reduced their participation because other students had done so. 

A. IMPACTS OF UP-FRONT DOCUMENTATION 

Nine of the districts included in the evaluation implemented UFD, whereby students 
applying for free or reduced-price meals had to provide documentation of their household 
circumstances (income/household size and/or receipt of food stamps or TANF benefits). The 
following subsections describe our findings on NSLP participation and participation/ 
reimbursement accuracy in UFD pilot and comparison districts. First, we give descriptive data 
on levels of participation in UFD pilot and comparison districts to provide a context for 
understanding the impact estimates, which we present next. We then give both descriptive and 
impact data on participation accuracy. Section A ends with a presentation of descriptive data on 
the two measures of reimbursement accuracy described in Chapter II. 

I. Descriptive Data on Levels of NSLP Participation 

Table III. la shows the estimated participation rates of various groups of students attending 
the UFD pilot and comparison districts. As described previously, these figures are estimates of 
the likelihood that a given student got a school lunch on a typical day, based on 

* Changes following implementation of the pilots do not necessarily mean that 
implementation caused the changes. Factors unrelated to the pilot may have caused these 
changes. 
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TABLE III. la 



NSLP PARTICIPATION RATES OE STUDENTS AS REPORTED BY PARENTS IN UP-ERONT 
DOCUMENTATION PILOT AND COMPARISON DISTRICTS, PALL 2002 



Probability That Student 
Obtained a School Lunch on a 
Given Day“ 

Comparison 

Student’s Certification/Eligibility Status Pilot Districts Districts 

Estimates Excluding Directly Certified Students 

All Students 



No school lunch (100-Pl) 


33.9 


27.4 




(1.48) 


(1.28) 


Any school lunch (PI) 


66.1 


72.6 




(1.48) 


(1.28) 


Pree lunch (P2) 


11.4 


13.6 




(0.65) 


(0.61) 


Reduced-price lunch (P3) 


3.9 


4.9 




(0.44) 


(0.39) 


Paid lunch (P4) 


50.5 


54.0 




(1.51) 


(1.35) 


Students Certified for Pree Meals (P_CP) 


91.3 


91.8 




(1.56) 


(1.95) 


Students Certified for Reduced-Price Meals (P_CRP) 


85.8 


87.6 




(3.43) 


(2.64) 


Students Not Certified for Pree or Reduced-Price Meals (P_NC) 


61.6 


68.1 




(1.69) 


(1.57) 


Students Eligible for Pree Meals (Income <130% PPL) 


No school lunch (100-P5) 


23.0 


16.5 




(2.71) 


(2.22) 


Any school lunch (P5) 


77.0 


83.5 




(2.71) 


(2.22) 


Pree lunch (P6) 


43.5 


50.0 




(2.96) 


(1.92) 


Reduced-price lunch (P7) 


5.3 


7.4 




(1.23) 


(0.73) 


Paid lunch (P8) 


28.1 


26.1 




(3.28) 


(2.42) 
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TABLE III. la (continued) 



Probability That Student 
Obtained a School Lunch on a 
Given Day“ 



Comparison 



Student’s Certification/Eligibility Status 


Pilot Districts 


Districts 


Students Eligible for Eree or Reduced-Price Meals (Income <185% PPL) 


No school lunch (100-P9) 


26.7 


17.5 




(2.65) 


(1.79) 


Any school lunch (P9) 


73.3 


82.5 




(2.65) 


(1.79) 


Eree lunch (PIO) 


29.7 


35.4 




(1.78) 


(1.35) 


Reduced-price lunch (PI 1) 


8.4 


10.7 




(1.13) 


(0.89) 


Paid lunch (PI 2) 


35.2 


36.3 




(2.75) 


(2.01) 


Students Not Eligible for Pree/Reduced-Price Meals (Income >185% PPL) 


No school lunch (100-P13) 


37.7 


32.4 




(1.87) 


(1.72) 


Any school lunch (PI 3) 


62.4 


67.6 




(1.87) 


(1.72) 


Eree lunch (PI 4) 


1.5 


1.7 




(0.38) 


(0.35) 


Reduced-price lunch (PI 5) 


1.9 


1.8 




(0.30) 


(0.27) 


Paid lunch (PI 6) 


59.0 


64.1 




(1.88) 


(1.74) 


Estimates for All Students, Including Those Directly Certified 

All Students 


No school lunch (100-Pl) 


33.6 


27.1 


Any school lunch (PI) 


66.4 


72.9 


Tree lunch (P2) 


13.3 


15.1 


Reduced-price lunch (P3) 


3.8 


4.8 


Paid lunch (P4) 


49.1 


53.0 


Students Certified for Tree Meals (P_CP) 


89.2 


90.2 


Students Certified for Reduced-Price Meals (P_CRP) 


85.8 


87.6 


Students Not Certified for Tree or Reduced-Price Meals (P_NC) 


61.6 


68.1 


Students Eligible for Eree Meals (Income <130% PPL) 


No school lunch (100-P5) 


22.3 


16.5 


Any school lunch (P5) 


77.7 


83.5 


Eree lunch (P6) 


49.2 


53.4 


Reduced-price lunch (P7) 


4.5 


6.6 


Paid lunch (P8) 


23.5 


23.4 
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TABLE III. la (continued) 



Probability That Student 
Obtained a School Lunch on a 
Given Day“ 



Comparison 

Student’s Certification/Eligibility Status Pilot Districts Districts 



Students Eligible for Eree or Reduced-Price Meals (Income <185% PPL) 



No school lunch (100-P9) 


25.8 


17.5 


Any school lunch (P9) 


74.2 


82.5 


Eree lunch (PIO) 


34.1 


38.3 


Reduced-price lunch (PI 1) 


7.5 


10.1 


Paid lunch (P12) 


31.7 


34.1 


Students Not Eligible for Pree/Reduced-Price Meals (Income >185% PPL) 


No school lunch (100-P13) 


37.7 


32.4 


Any school lunch (PI 3) 


62.4 


67.6 


Eree lunch (PI 4) 


1.5 


1.7 


Reduced-price lunch (PI 5) 


1.9 


1.8 


Paid lunch (PI 6) 


59.0 


64.1 



Note; This table is intended to provide descriptive information on participation rates in UPD pilot and comparison 
districts only. Differences between participation rates in pilot and comparison districts should not be 
interpreted as estimates of the impact of UPD on NSLP participation rates. See Table III.2 for impact 
estimates. 

“Standard errors are in parentheses. These standard errors have been adjusted to account for the complex sampling 
design of the data set. 

PPL = federal poverty level. 
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a parent’s report of the frequency of the student’s participation. The top panel shows 
participation rate estimates for students in the pilot and comparison districts who were not 
directly certified; the bottom panel shows estimates for all students, including those directly 
certified. 

The discussion below focuses initially on participation rates in UFD comparison districts. 
This provides a sense of the counterfactual — what the situation would have been like in the pilot 
districts without the UFD intervention. We then discuss how participation rates differ in pilot 
and comparison districts. While the difference between pilot and comparison districts’ 
participation rates suggests the likely effect of the pilot on participation, the estimates in Table 
III. la do not attempt to control for student characteristics. Thus, the pilot-comparison difference 

may be due to differences in the characteristics of the two groups of students rather than to the 

2 

effect of the pilot intervention. 

According to parents’ reports in UFD comparison districts, 72.6 percent of students — 
excluding those directly certified — got a school lunch on a typical day in fall 2002 (Table 
III. la). This overall participation rate includes 13.6 percent of students who got a free lunch, 
4.9 percent who got a reduced-price lunch, and 54.0 percent who got a paid lunch. Based on 
parents’ reports, just over one-quarter of students did not get a school lunch on a typical day. 

Among non-directly certified students who were certified for free or reduced-price meals, 
about 9 of 10 were reported to have obtained a school lunch on a typical day (91.8 percent 

2 

See Subsection A. 2 for estimates of the impacts of UFD in which we do attempt to control 
for differences in the characteristics of the two groups of students. 

-5 

Although most of the discussion here focuses on the sample of students for which we 
collected survey data, which excludes directly certified students. Table III. la (as well as Table 
III. lb. Table III.3, and each of the corresponding GV tables) also includes estimates for all 
students, including those directly certified. 
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among those certified for free meals and 87.6 percent among those certified for reduced-price 
meals). Approximately two-thirds of noncertified students got a school lunch on a typical school 
day. 

We also examined participation rates separately among non-directly certified students with 
family incomes that make them eligible for free/reduced-price meals (or free meals alone) and 
those not income-eligible for benefits. Participation rates were higher among the lower- income 
groups than among the higher-income one. Among non-directly certified students income 
eligible for free meals, for example, 83.5 percent got a school lunch on a typical day, including 
50.0 percent who got a free lunch, 7.4 percent who got a reduced-price lunch, and 26.1 percent 
who got a paid lunch (Table III. la). A similar pattern is evident among students with incomes 
less than 185 percent of the FPL — and thus eligible for free or reduced-price meals — with 
participation rates of 82.5 percent overall, 35.4 percent for free lunches, 10.7 percent for 
reduced-price lunches, and 36.3 percent for paid lunches. 

Table III. lb renormali z es the participation rates in Table III. la to show the implicit 
percentages of all meals served that are in each reimbursement category, thus providing 
additional perspective on the participation patterns of students within each income subgroup. 
Among all lunches served to non-directly certified students with incomes below 130 percent of 
poverty in UFD comparison districts, for example, 59.9 percent were free, while 8.8 percent 
were reduced-price, and the remaining 31.3 percent were paid (Table III. lb). These percentages 
show that substantial numbers of meals to low-income children are being served as paid meals. 
In particular, nearly one-third of the reimbursable lunches served to non-directly certified 
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TABLE III. lb 



REIMBURSEMENT CATEGORY OE MEALS RECEIVED BY NSLP PARTICIPANTS IN UP-ERONT 
DOCUMENTATION PILOT AND COMPARISON DISTRICTS, PALL 2002 



Percentage Distribution of 
Participants by Reimbursement 
Category’’ 

Comparison 

Student’s Estimated Eligibility Status’* Pilot Districts Districts 



Estimates Excluding Directly Certified Students 

Students Eligible for Pree Meals (Income <130% PPL) 



Pree lunch (P6 / P5) 


56.5 


59.9 




(3.72) 


(2.38) 


Reduced-price lunch (P7 / P5) 


6.9 


8.8 




(1.59) 


(0.88) 


Paid lunch (P8 / P5) 


36.6 


31.3 




(3.91) 


(2.44) 


Students Eligible for Pree or Reduced-Price Meals (Income <185% PPL) 


Pree lunch (PIO / P9) 


40.6 


43.0 




(2.50) 


(1.70) 


Reduced-price lunch (P 1 1 / P9) 


11.5 


13.0 




(1.56) 


(1.10) 


Paid lunch (P12/P9) 


47.9 


44.0 




(2.80) 


(1.89) 


Students Not Eligible for Pree/Reduced-Price Meals (Income >185% PPL) 


Pree lunch (P14/P13) 


2.5 


2.5 




(0.61) 


(0.53) 


Reduced-price lunch (P15 / P13) 


3.0 


2.7 




(0.49) 


(0.40) 


Paid lunch (P16/P13) 


94.5 


94.8 




(0.77) 


(0.66) 


Estimates for All Students, Including Those Directly Certified 

Students Eligible for Pree Meals (Income <130% PPL) 


Pree lunch (P6 / P5) 


63.3 


64.4 


Reduced-price lunch (P7 / P5) 


6.1 


7.9 


Paid lunch (P8/P5) 


30.7 


27.6 


Students Eligible for Pree or Reduced-Price Meals (Income <185% PPL) 


Pree lunch (P10/P9) 


45.9 


46.7 


Reduced-price lunch (PI 1 / P9) 


10.2 


12.2 


Paid lunch (P12/P9) 


43.9 


41.0 



PPL = federal poverty level. 

“NSLP participation is reported by parents in the study survey. 

’’Standard errors are in parentheses. These standard errors have been adjusted to account for the complex sampling 
design of the data set. 
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students who were income eligible for free meals were full-price (or paid) lunches.'* Among 
lunches served to students with incomes of no more than 185 percent of poverty, on the other 
hand, 44.0 percent were full price. Among lunches served to higher-income students, most (94.8 
percent) were paid, with 2.5 percent free and 2.7 percent reduced-price. 

Tables III. la and III. lb also show participation rates in UFD pilot districts. For nearly all of 
the measures shown in Table III. la, the participation rate in the pilot district is lower than the 
corresponding rate in the comparison district. For example, the overall participation rate among 
non-directly certified students reported by parents in UFD pilot districts was 66.1 percent, 
compared with 72.6 percent in comparison districts. The free, reduced-price, and paid 
participation rates among all non-directly certified students were also lower in pilot districts than 
in comparison ones. 

2. Impacts on Participation Rates 

Comparisons of the differences in the participation rates reported in Tables III. la and III. lb 
in UFD pilot and comparison districts do not take into account differences in the characteristics 
of students who attend the pilot and comparison districts. Thus, although the participation rates 
in pilot districts are lower than those in comparison districts, this could be due either to 
differences in the student or district characteristics or to the effects of the UFD intervention. As 
described in Chapter II, we estimated regression models to control for the observable 
characteristics of students and their families before estimating the impacts of UFD (including 
characteristics such as race/ethnicity and educational attainment of the household head). The 
regression models do not directly control for the effects of district-level differences that could 

When directly certified students are included in this calculation, the paid participation rate 
among students eligible for free meals is 27.6 percent (Table III. lb). 
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influence individual participation decisions. Our analytical approach has attempted to control for 
these types of differences by selecting a comparison district for each pilot district that matches 
the pilot on a range of factors that could affect certification and participation (such as the 
districts’ baseline certification rates and poverty rates). ^ 

Table III.2 shows the results of the impact analysis. The first two columns of the table show 
the regression-adjusted participation rates in pilot and comparison districts. These participation 
rates have been adjusted to take into account the fact that students in pilot and comparison 
districts may differ in some characteristic that influences their participation. To the extent that 
the regression model has controlled for all relevant characteristics, the remaining difference 
between pilot and comparison districts in the regression- adjusted participation rate can be 
attributed to the impact of UFD pilot procedures. Thus, the final column of the table shows the 
estimate of the impact, which is the difference between the regression-adjusted mean in pilot 
versus comparison districts. 

UFD is estimated to lead to a reduction in the reported overall NSLP participation rate 
(among non-directly certified students) of 5.8 percentage points, from a regression-adjusted rate 
of 72.3 percent in comparison districts to 66.5 percent in pilot districts (Table III.2). This 
difference is statistically significant at the 0.01 level. 

Conceptually, the change in the overall NSLP participation rate can be decomposed into the 
change in the free or reduced-price lunch participation rate and the paid participation rate. About 
half of the overall change is due to reductions in free or reduced-price meal participation, as the 
impact on this participation rate is -2.8 percentage points and is statistically significant. The rest 



^ See Burghardt et al. (2004) for a description of the comparison district matching process. 
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TABLE III.2 



ESTIMATED IMPACT OE UP-ERONT DOCUMENTATION PILOT PROJECTS ON NSLP PARTICIPATION 
(AS REPORTED BY PARENTS) AMONG STUDENTS NOT DIRECTLY CERTIEIED, EALL 2002 





Regression-Adjusted Rate“ 




Measure of Participation 


Pilot 


Comparison 


Impact’’ 


All Students 








Any school lunch (PI) 


66.5 


72.3 


-5.8** 

(2.07) 


Eree lunch (P2) 


11.5 


13.6 


-2.1* 

(1.05) 


Eree or reduced-price lunch (P2 h-P3) 


15.6 


18.4 


-2.8* 

(1.20) 


Paid lunch (P4) 


50.6 


53.8 


-3.3 

(2.22) 


Students Eligible for Eree Meals (Income 
<130% EPL) 








Any school lunch (P5) 


77.6 


82.9 


-5.4 

(3.77) 


Eree lunch (P6) 


43.6 


49.6 


-6.0 

(4.24) 


Eree or reduced-price lunch (P6 h-P7) 


49.1 


56.9 


-7.8 

(4.30) 


Paid lunch (P8) 


27.8 


26.0 


1.8 








(4.69) 


Students Eligible for Eree or Reduced-Price 
Meals (Income <185% PPL) 








Any school lunch (P9) 


73.1 


82.8 


-9.8** 

(3.30) 


Eree lunch (PIO) 


29.3 


35.6 


-6.2* 

(3.10) 


Eree or reduced-price lunch (PlO-tPll) 


37.4 


46.7 


_9 3** 

(3.37) 


Paid lunch (P12) 


34.8 


36.1 


-1.4 

(4.16) 


Students Not Eligible for Pree/Reduced-Price 
Meals (Income >185% PPL) 








Any school lunch (PI 3) 


62.8 


67.2 


- 4.4 

(2.50) 


Eree lunch (PI 4) 


1.6 


1.7 


-0.1 

(0.62) 


Eree or reduced-price lunch (P14H-P15) 


3.6 


3.4 


0.2 








(0.75) 


Paid lunch (PI 6) 


59.2 


63.8 


- 4.6 

( 2 . 55 ) 
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TABLE III. 2 {continued) 



FPL = federal poverty level. 

“The regression-adjusted rate shows the estimated participation rate that would result under the assumption that 
students in both UFD pilot and comparison districts had the same distribution of observable characteristics. Thus, 
any remaining difference between the regression-adjusted participation rate in pilot versus comparison district can 
be attributed to the impact of the pilot intervention. 

’’Standard errors are in parentheses. These standard errors have been adjusted to account for the complex sampling 
design of the data set. 

*Significantly different from zero at the .05 level, two-tailed test. 

**Significantly different from zero at the .01 level, two-tailed test. 
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of the overall impact comes from a reduction in the paid participation rate, which is estimated to 



decrease by 3.3 percentage points, although this impact is not statistically significant at the 
0.05 level.® 

Among students income eligible for free meals, the regression-adjusted overall participation 
rate was 5.4 percentage points lower in pilot districts than in comparison ones (77.6 versus 
82.9 percent). This difference is not statistically significant at the 0.05 level (Table III.2). This 
pilot-comparison difference arises essentially from a difference in the free/reduced-price 
participation rate, which was 7.8 percentage points lower in pilot districts than in comparison 
ones (49.1 versus 56.9 percent; again, not significant at the 0.05 level). The paid lunch 
participation rate among students who were income eligible for free meals was similar in pilot 
and comparison districts. 

The patterns of estimated impacts were similar among students income eligible for free or 
reduced-price meals (that is, with incomes below 185 percent of poverty), except that the impacts 
were statistically significant. UFD was estimated to have led to a statistically significant 
9.8 percentage point decline in the overall NSLP participation rate of non-directly certified 
students in this income category, from 82.8 to 73.1 percent (Table III.2). This impact was driven 



^Because of concerns about the quality of the participation data reported by parents, we 
conducted several sensitivity tests to determine whether the estimated impacts of UFD on the 
participation rate changed when we changed our data, our methods, or our sample. Appendix A 
describes the estimated impacts of UFD on participation rates based on administrative data. In 
addition, we conducted two additional sensitivity tests. First, we re-estimated the impact models 
after excluding one site in which the quality of the parent-report participation data appeared 
particularly suspect. The exclusion of this site did not influence the estimated impacts of UFD 
on NSLP participation rates. Second, we re-estimated the model after excluding all high school 
students, under the reasoning that the parents of high school students would be less likely to 
know their children’s NSLP participation status than the parents of younger students. The 
exclusion of this group did not substantially influence the key estimates from the model — the 
estimated impact of UFD on free and reduced-price NSLP participation. 
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almost entirely by a decrease in the percentage of students getting free or reduced-price meals, as 
the impact on the free/reduced-price participation rate was -9.3 percentage points, and 
statistically significant. In contrast, the estimated impact on the paid participation rate among 
this group of students income eligible for free or reduced-price meals was -1.4 percentage 
points, and not statistically significant. 

This pattern of impacts among students income eligible for free or reduced-price meals is 
consistent with the estimated impact of UFD on rates of certification among this same group, 
presented in Burghardt et al. (2004). That study found that UFD led to a statistically significant 
decline in the free/reduced-price certification rate among students with incomes of no more than 
185 percent of poverty. Thus, it is consistent that the pilot would also lead to a decline in the 
percentage of this low-income group who got free or reduced-price meals. The UFD pilot had 
little or no impact on paid meal participation within this low-income group. 

Among students not income eligible for free or reduced-price meals (that is, with incomes of 
more than 185 percent of poverty), the estimated impact of UFD on the overall participation rate 
was -4.4 percentage points (P13), from a regression-adjusted rate of 67.2 percent in comparison 
districts to 62.8 percent in pilot districts (Table III.2). Since nearly all higher-income 
participants got paid lunches rather than free or reduced-price lunches, this impact was due to an 
impact of a similar size (-4.6 percentage points) on the paid participation rate (P16). However, 
neither of these estimated impacts was statistically significant at the 0.05 level. 

According to Burghardt et al. (2004), the estimated impact of the UFD pilots on the 
certification of students from households with incomes exceeding 185 percent of poverty was 
zero: 3.9 percent of this income group was certified in comparison sites, and 3.9 percent was 
certified in pilot sites. Consistent with this, we found that, among this higher-income group, the 
percentage getting free or reduced-price meals was nearly identical in UFD pilot and comparison 
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districts. However, we also found that the estimated regression- adjusted paid participation rate 
(PI 6) among students from households with incomes exceeding 185 percent of poverty was 
almost five percentage points lower in UFD pilot districts than in comparison ones. What could 
account for the fact that higher-income students were estimated to be less likely to get paid 
lunches in pilot districts, given that the pilot intervention focused on the certification process? 

Sampling variability is one possible explanation. In fact, the estimated impact on the paid 
participation rate was not statistically significant at the 0.05 level. Despite this lack of 
significance, however, we believe that the estimated impact of UFD on paid participation among 
higher-income students is worth examining, for two reasons. First, this impact estimate would 
have been statistically significant if we had used a 0.10 threshold for statistical significance. In 
other words, while the null hypothesis of no impact cannot be rejected with 95 percent 
confidence, it can be rejected with more than 90 percent confidence. 

A second piece of evidence that lends credence to a negative impact of UFD on paid 
participation among higher-income students is that an examination of administrative data 
provides similar evidence. In particular, we conducted an analysis of aggregate administrative 
data on participation in which we examined changes in participation rates between the prepilot 
baseline period and the 2002-2003 follow-up period (see Appendix A). In this analysis, the pilot 
districts themselves (in the baseline period) essentially served as their own control group. The 
analysis resulted in findings on the impacts of the pilot on overall participation and participation 



An alternative way of presenting the estimated impact of UFD on the paid participation 
rate among higher-income students is through a 95 percent confidence interval. This confidence 
interval indicates that we are 95 percent confident that the true impact falls within the range 
-9.60 to 0.04 percentage points. 
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by meal price category that were similar to the findings based on the regression model and 
presented above. 

If the negative impact of UFD on the paid participation rate of higher- income students (PI 6) 
is real and not the result of sampling variability, then what factors could explain it? As noted 
above, UFD was estimated to lead to no change in the proportion of students who were certified 
for free or reduced-price meals (and thus no change in the proportion not certified) among those 
from households with incomes exceeding 185 percent of poverty. Thus, the negative impact on 
the paid participation rate (PI 6) among these higher- income students had to come about through 
a change in the behavior of (noncertified) students whose certification status was not affected by 
UFD. We do not have direct evidence on what might have led to this negative impact. 
However, one possible explanation is a peer effect. Given our finding that UFD led to a decrease 
in participation among lower-income students (P9), the peer effect explanation would imply that, 
because of reduced participation among their peers, noncertified higher-income students may 
have chosen to obtain a school lunch less frequently than they would have otherwise. 

3. Descriptive Data on Levels of Participation Accuracy 

To examine the impacts of UFD on the accuracy with which students receive free or 
reduced-price meals, we estimated four alternative measures of free or free/reduced-price 
participation accuracy that correspond to the measures of certification accuracy presented in 
Burghardt et al. (2004). We describe these measures in Chapter II. Each is a measure of the 
proportion of meals served free or free/reduced-price that went to students whose incomes made 
them eligible for these benefits. We estimated a single measure of free meal accuracy — the 
percentage of free meals served to students with incomes of no more than 130 percent of 
poverty. 
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Due to our sampling strategy of excluding directly certified students, each of these measures 
of participation accuracy in the top panel of Table III.3 is based only on those students certified 
for free or free/reduced-price meals by application. The bottom panel includes estimates of the 
participation accuracy among all students getting free meals or getting free or reduced-price 
meals, including directly certified students. The latter set of estimates relies on the assumption 
that directly certified students who get school lunches are eligible for free meals. This 
assumption is based on previous research showing that more than 9 out of every 10 directly 
certified students are income eligible for the free meal benefits they are receiving (see, for 
example. Food and Nutrition Service 2002; and Gleason et al. 2003). Thus, the estimated levels 
of participation accuracy among non-directly certified students reported in this chapter are lower 
than levels of participation accuracy among all students shown in the bottom panel of the table. 

The free meal accuracy rate among non-directly certified students in UFD comparison 

o 

districts was 78.4 percent in fall 2002 (Table III.3). This implies that, among free meals served 
to non-directly certified students in these districts, just under four out of five were served to 
students eligible for this benefit — that is, those with incomes of no more than 130 percent of 
poverty. Conversely, 21.6 percent of free meals obtained by students went to those with incomes 
above 130 percent of poverty. Accuracy rates for all free meals, including those served to 
directly certified students, are about four percentage points higher, as shown in the bottom panel 
of the table. 



However, we also estimated that, among students getting free school lunches in UFD 
comparison districts, 91.1 percent are income eligible for either free or reduced-price meals. 
Only 8.9 percent of free meal participants, in other words, are ineligible for both free and 
reduced-price meals. 
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TABLE III.3 



PARTICIPATION ACCURACY AMONG STUDENTS IN UP-ERONT DOCUMENTATION 
PILOT AND COMPARISON DISTRICTS, PALL 2002 





Estimated Rate’’ 


Participation Accuracy Measures” 


Pilot Districts 


Comparison 

Districts 


Estimates Excluding Directly Certified Students 






Pree Meal Accuracy 

PA_P: Percentage of Pree Meals Served to Students Income Eligible 
for Pree Meals” 


80.0 

(2.86) 


78.4 

(3.09) 


Pree/Reduced-Price Meal Accuracy 

PA_PRP1; Percentage of Pree or Reduced-Price Meals Served to 
Students Eligible for the Level of Benefits They Are Receiving 


71.3 

(2.66) 


69.2 

(2.58) 


PA_PRP2; Percentage of Pree or Reduced-Price Meals Served to 
Students Not Overcertified for Benefits They Are Receiving 


78.5 

(2.40) 


77.9 

(2.49) 


PA_PRP3: Percentage of Pree or Reduced-Price Meals Served to 
Students with Incomes Below 185% of the PPL 


85.6 

(1.98) 


87.0 

(1.95) 


Estimates for All Students, Including Those Directly Certified 






Pree Meal Accuracy 

PA_P: Percentage of Pree Meals Served to Students Income Eligible 
for Pree Meals 


85.1 


82.0 


Pree/Reduced-Price Meal Accuracy 

PA_PRP1: Percentage of Pree or Reduced-Price Meals Served to 
Students Eligible for the Level of Benefits They Are Receiving 


76.8 


72.7 


PA_PRP2; Percentage of Pree or Reduced-Price Meals Served to 
Students Not Overcertified Benefits They Are Receiving 


82.8 


80.4 


PA_PRP3; Percentage of Pree or Reduced-Price Meals Served to 
Students with Incomes Below 185% of the PPL 


88.6 


88.4 



PPL = federal poverty level. 



Note; This table is intended to provide descriptive information on participation accuracy rates in Up-Pront 
Documentation (UPD) pilot and comparison districts only. Differences between accuracy rates in pilot 
and comparison districts should not be interpreted as estimates of the impact of UPD on NSLP 
participation accuracy. See Table III.4 for impact estimates. 

“NSLP participation is reported by parents in the study survey. 

’’Standard errors are in parentheses. These standard errors have been adjusted to account for the complex sampling 
design of the data set. 

“Among students who are served free meals, 89.9 percent of those in pilot districts and 91.1 percent of those in 
comparison districts are income eligible for either free or reduced-price meals. 
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The estimated free/reduced-price meal accuracy rate was in the same general range, 
although its value depended on the definition of accuracy used. According to the most stringent 
definition of accuracy, among non-directly certified children 69.2 percent of free or reduced- 
price meals served went to those who were income eligible for exactly the benefit level they 
received. In contrast, 77.9 percent of these meals were served to students who were not 
overcertified, and 87.0 percent were served to those who were income eligible for either free or 
reduced-price meals. 

Before controlling for any student characteristics, participation accuracy rates were similar 
in UFD pilot districts to those in comparison districts (as described above). For each of the four 
accuracy measures, the participation accuracy rate among non-directly certified students in pilot 
districts was about two percentage points (or less) different from the comparable accuracy rate in 
comparison districts. 

4. Impacts on Participation Accuracy 

Using the methodology described in Chapter II, we estimated the impacts of UFD on each of 
the four measures of participation accuracy that exclude directly certified students. These 
impacts reflect the difference in the predicted accuracy rate after controlling for student 
characteristics. Table III. 4 shows these regression-adjusted participation accuracy rates in pilot 
and comparison districts, along with the estimated impact of the pilot on participation accuracy. 

For each of the four measures of participation accuracy, the estimated impact of UFD was 
not statistically significant. For example, the regression-adjusted free meal accuracy rate in pilot 
districts was 76.6 percent, 1.7 percentage points lower than the rate of 78.3 percent in 
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TABLE III.4 



ESTIMATED IMPACT OE UP-ERONT DOCUMENTATION ON PARTICIPATION ACCURACY 
AMONG STUDENTS NOT DIRECTLY CERTIEIED, EALL 2002 





Regression-Adjusted Rate’’ 




Participation Accuracy Measures” 


Pilot 


Comparison 


ImpacL 


Tree Meal Accuracy 








PA_E; Percentage of Tree Meals Served to 
Students Income Eligible for Eree Meals 


76.6 


78.3 


-1.7 

(10.86) 


Eree/Reduced-Price Meal Accuracy 








PA_ERP1 : Percentage of Eree or Reduced- 

Price Meals Served to Students Eligible for the 
Level of Benefits They Are Receiving 


67.5 


69.8 


-2.3 

(8.30) 


PA_ERP2: Percentage of Eree or Reduced-Price 
Meals Served to Students Not Overcertified for 
the Level of Benefits They Are Receiving 


72.6 


79.4 


-6.76 

(8.82) 


PA_ERP3: Percentage of Eree or Reduced-Price 
Meals Served to Students with Incomes Below 
185% of the PPL 


78.6 


88.8 


-10.2 

(8.04) 



EPL = federal poverty level. 

“NSLP participation is reported by parents in the study survey. 

’’The regression-adjusted rate shows the estimated participation accuracy rate that would result under the assumption 
that students in both UED pilot and comparison districts had the same distribution of observable characteristics. 
Thus, any remaining difference between the regression-adjusted accuracy rate in pilot versus comparison district 
can be attributed to the impact of the pilot intervention. 

‘’Standard errors are in parentheses. These standard errors have been adjusted to account for the complex sampling 
design of the data set. 

*Significantly different from zero at the .05 level, two-tailed test. 

**Significantly different from zero at the .01 level, two-tailed test. 
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comparison districts (Table III.4). As the standard error of this estimated impact was much 
greater than 1.7 percentage points, the estimated impact was not statistically significant. 

The estimated impacts of UFD on each of the three measures of free/reduced-price 
participation accuracy were also not statistically significant. In each case, the standard error of 
the impact estimate was quite large, so we could not be confident the observed difference 
between pilot and comparison districts was not due to chance. In addition, the magnitude of the 
estimated impact was somewhat sensitive to the measure of free/reduced-price participation 
accuracy used. In particular, the estimated impacts of UFD ranged from -2.3 to -10.2 
percentage points. 

Given that each of the estimated impacts of UFD on participation accuracy was not 
statistically significant, we cannot conclude that the pilot had any influence on the accuracy with 
which free or free/reduced-price lunches go to students. However, the fact that the standard error 
of each impact estimate was large and the magnitude of the impact estimate was sensitive to the 
definition of accuracy used suggests that the methodology we used led to unstable results. 
Further evidence of the instability of the estimated impacts on participation accuracy came from 
another sensitivity test we conducted. Because the quality of information on students’ NSLP 
participation as reported by parents appears especially suspect for students in one particular 
district, we reestimated each of our impact models after excluding this district.^ Although the 



^ See Appendix A for evidence on parents’ reporting of their children’s participation status 
in each UFD and GV district, including the district for which estimated participation rates based 
on parents’ reports is greatly different from the estimates based on administrative data. 
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exclusion of this site did not materially influence the estimated impact of UFD on rates of 



participation, it affected the estimated impact on rates of participation accuracy.'*^ 

5. Reimbursement Accuracy 

Chapter II defined two measures of the extent to which federal reimbursements for free, 
reduced-price, or paid meals are either higher than or lower than they should be according to the 
survey-based free/reduced-price eligibility status of the certified students who consumed meals. 
The first captures payments in excess of what districts should receive based on students’ 
eligibility status. The second captures the shortfall in reimbursement payments due to students 
who are actually certified for one level of benefits (reduced-price meals) when they should be 
certified for a higher level of benefits (free meals) based on their eligibility status. The 
calculations presented here assume that the certification status change would not affect the 
participation behavior of students whose certification status was changed in this way. 

Among reimbursements for all meals served to non-directly certified students in UFD 
comparison districts, we estimated that 7.7 percent are “overpayments” in the sense of being paid 
for meals served to students who are certified for a higher level of benefits than that for which 
they are eligible (Table III.5). In other words, if the certification status of all students who were 
certified for a higher level of benefits than they were eligible for was suddenly adjusted and they 
became certified for the appropriate level of benefits given their income, and if their participation 
behavior did not change as a result of the certification status change, the federal reimbursements 



In particular, the magnitude of the estimated impacts on participation accuracy changed 
substantially when this site was excluded. However, the results did not lead to any changes in 
the levels of statistical significance of the estimated impacts on participation accuracy. In both 
the full sample and in the sample that excluded the suspect site, each of these impact estimates 
was not statistically significant. 
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TABLE III.5 



FEDERAL REIMBURSEMENTS FOR LUNCHES RECEIVED BY STUDENTS WHO ARE NOT DIRECTLY 
CERTIFIED IN UP-FRONT DOCUMENTATION PILOT AND COMPARISON DISTRICTS, FALL 2002 





Estimated Rate’’ 


Reimbursement Measures“ 


Pilot Districts 


Comparison 

Districts 


RA_OP: Percentage of All Federal Reimbursements for Meals Served to 
Students Not Eligible for the Level of Benefit They Are Receiving 


7.3 

(0.9) 


7.7 

(1.1) 


RA_UP: Percentage Short-Fall in Federal Reimbursements Because Some 
Certified Students Received a Lower Benefit Than They Were Eligible for 


0.8 

(0.2) 


1.1 

(0.1) 



Note: This table is intended to provide descriptive information on reimbursement accuracy in Up-Front 

Documentation (UFD) pilot and comparison districts only. Differences between rates in pilot and 
comparison districts should not be interpreted as estimates of the impact of UFD on reimbursement 
accuracy. 

“Reimbursements are calculated by using student participation as reported by parents in the study survey and 
applying the reimbursement appropriate for the student’s meal price status. 

’’Standard errors are in parentheses. These standard errors have been adjusted to account for the complex sampling 
design of the data set. 
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to the comparison districts would decline by 7.7 percent. The analogous rate of reimbursements 
in excess of students’ eligibility status in UFD pilot districts was similar, at 7.3 percent. 

A different measure of the extent to which federal reimbursements are not accurate involves 
examining the extent to which certified students are undercertified — that is, are receiving 
reduced-price meals when they should be receiving free meals. This turns out to have a 
relatively small effect on the overall level of reimbursements. If the eligibility status of all of 
these undercertified students was suddenly changed so that their benefits were increased to the 
appropriate level given their income, federal reimbursements in UFD comparison districts would 
have increased by only 1.1 percent (Table III.5).'* Again, the comparable rate in UFD pilot 
districts was similar: 0.8 percent. 



B. IMPACTS OF GRADUATED VERIFICATION 

Three of the districts in the evaluation implemented GV. Districts using GV increased the 
scale of their verification efforts if their initial verification of approved applications resulted in 
large proportions of verified households whose benefits were reduced or terminated. Thus, these 
additional verification efforts took place — if they took place at all — only after the initial 
verification efforts were completed and the results determined. Since data collection for the 
study was conducted primarily during October through January, it is important to note that the 
outcomes measured in this study reflect conditions in GV districts prior to the completion of any 



** This ignores another type of undercertification — students not certified for either free or 
reduced-price meals when they are eligible for these benefits. That situation could have resulted 
from one of three scenarios: (1) the household could have applied for benefits and been denied, 
(2) the household could have decided not to apply for benefits (or not to complete the application 
process) because of the requirements the pilot imposed, or (3) the student could have decided not 
to apply for benefits because he or she did not wish to consume school meals at all or meals that 
were free or reduced-price. Because we could not fully distinguish between these scenarios and 
did not wish to combine them in our analysis, we did not analyze this type of undercertification. 
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additional verification activities that took place in the district. Correspondingly, the measured 
impacts of GV described in this section primarily reflect the cumulative effects of the GV 
activities that took place in prior years, and not the effects of GV activities during the year in 
which the data were collected (year 3 of the pilot intervention). 

1. Descriptive Data on Levels of NSLP Participation 

According to parents’ reports in GV comparison districts, 81.5 percent of non-directly 

1 2 

certified students got a school lunch on a typical day in fall 2002 (Table III.6a). Among all 
these non-directly certified students, 24.2 percent got a free lunch, 9.3 percent got a reduced- 
price lunch, and 47.9 percent got a paid lunch. Based on parents’ reports, fewer than one in five 
students did not eat a school lunch on a typical day. 

As in UFD comparison districts, about 9 of 10 students in GV comparison districts who 
were certified for free (by application) or for reduced-price meals got a school lunch on a typical 
school day. In particular, the participation rate was 93.0 percent among those certified for free 
meals and 89.1 percent among those certified for reduced-price meals. Three-fourths of 
noncertified students got a school lunch on a typical day. 

Participation rates were also higher among the lower-income students than among the 
higher- income ones in GV comparison districts. Among non-directly certified students eligible 
for free meals (that is, with incomes of no more than 130 percent of poverty), for example, 92.2 
percent got a school lunch on a typical day, including 64.5 percent who got a free lunch (Table 
III.6a). The overall participation rates among students eligible for free or reduced-price meals 

1 2 

The GV participation rate estimates that include all students are shown in the bottom 
panel of Table III.6b. Among all students in GV comparison districts, an estimated 82.8 percent 
get a school lunch on a typical day (according to parents’ reports). Administrative data suggests 
that the participation rate in GV comparison districts is 62.3 percent (see Appendix A). 
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TABLE III.6a 



NSLP PARTICIPATION RATES OE STUDENTS AS REPORTED BY PARENTS IN GRADUATED 
VERIEICATION PILOT AND COMPARISON DISTRICTS, PALL 2002 



Probability That Student 
Obtained a School Lunch on a 
Given Day^* 

Comparison 

Student’s Certification/Eligibility Status Pilot Districts Districts 

Estimates Excluding Directly Certified Students 

All Students 



No school lunch (100 -PI) 


22.1 


18.5 




(1.84) 


(1.78) 


Any school lunch (PI) 


77.9 


81.5 




(1.84) 


(1.78) 


Pree lunch (P2) 


20.8 


24.2 




(1.22) 


(1.40) 


Reduced-price lunch (P3) 


7.2 


9.3 




(0.83) 


(1.11) 


Paid lunch (P4) 


49.4 


47.9 




(2.09) 


(2.00) 


Students Certified for Pree Meals (P_CP) 


91.7 


93.0 




(3.71) 


(1.26) 


Students Certified for Reduced-Price Meals (P_CRP) 


91.9 


89.1 




(3.71) 


(4.08) 


Students Not Certified for Pree or Reduced-Price Meals (P_NC) 


71.7 


75.0 




(2.49) 


(2.63) 


Students Eligible for Pree Meals (Income <130% PPL) 


No school lunch (100 -P5) 


15.8 


7.8 




(3.33) 


(1.76) 


Any school lunch (P5) 


84.2 


92.2 




(3.33) 


(1.76) 


Pree lunch (P6) 


51.4 


64.5 




(3.98) 


(3.95) 


Reduced-price lunch (P7) 


6.3 


10.7 




(2.17) 


(1.47) 


Paid lunch (P8) 


26.4 


16.9 




(4.28) 


(3.88) 
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Table III.6a (continued) 



Probability That Student 
Obtained a School Lunch on a 
Given Day“ 



Comparison 



Student’s Certification/Eligibility Status 


Pilot Districts 


Districts 


Students Eligible for Eree or Reduced-Price Meals (Income <185% PPL) 


No school lunch (100 -P9) 


14.0 


10.2 




(2.41) 


(1.85) 


Any school lunch (P9) 


86.0 


89.8 




(2.41) 


(1.85) 


Eree lunch (PIO) 


42.2 


50.6 




(2.76) 


(2.66) 


Reduced-price lunch (Pll) 


13.3 


15.9 




(2.18) 


(1.32) 


Paid lunch (P12) 


30.0 


23.3 




(3.38) 


(2.84) 


Students Not Eligible for Pree/Reduced-Price Meals (Income >185% PPL) 


No school lunch (100 -PI 3) 


29.2 


26.5 




(2.78) 


(2.83 ) 


Any school lunch (P 1 3) 


70.8 


73.6 




(2.78) 


(2.83) 


Eree lunch (PI 4) 


1.6 


2.6 




(0.64) 


(0.64) 


Reduced-price lunch (PI 5) 


2.7 


6.4 




(0.35) 


(2.09) 


Paid lunch (PI 6) 


66.5 


64.6 




(2.81) 


(3.00) 


Estimates for All Students, Including Those Directly Certified 

All Students 


No school lunch (100 -PI) 


20.4 


17.2 


Any school lunch (PI) 


79.6 


82.8 


Tree lunch (P2) 


29.5 


31.7 


Reduced-price lunch (P3) 


6.4 


8.2 


Paid lunch (P4) 


43.4 


42.8 


Students Certified for Tree Meals (P_CP) 


92.2 


92.7 


Students Certified for Reduced-Price Meals (P_CRP) 


91.9 


89.1 


Students Not Certified for Tree or Reduced-Price Meals (P_NC) 


71.7 


75.0 


Students Eligible for Eree Meals (Income <130% PPL) 


No school lunch (100 -P5) 


12.3 


7.8 


Any school lunch (P5) 


87.7 


92.2 


Eree lunch (P6) 


66.5 


73.5 


Reduced-price lunch (P7) 


4.2 


7.0 


Paid lunch (P8) 


16.9 


11.6 
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Table III.6a (continued) 



Probability That Student 
Obtained a School Lunch on a 
Given Day“ 



Comparison 



Student’s Certification/Eligibility Status 


Pilot Districts 


Districts 


Students Eligible for Free or Reduced-Price Meals (Income <185% FPL) 


No school lunch (100 -P9) 


12.3 


9.7 


Any school lunch (P9) 


87.7 


90.3 


Free lunch (PIO) 


54.7 


59.8 


Reduced-price lunch (Pll) 


10.1 


12.4 


Paid lunch (P12) 


22.5 


18.0 


Students Not Eligible for Eree/Reduced-Price Meals (Income >185% PPL) 


No school lunch (100 -PI 3) 


29.2 


26.5 


Any school lunch (P 1 3) 


70.8 


73.6 


Free lunch (PI 4) 


1.6 


2.6 


Reduced-price lunch (PI 5) 


2.7 


6.4 


Paid lunch (PI 6) 


66.5 


64.6 



FPL = federal poverty level. 

Note; This table is intended to provide descriptive information on participation rates in Graduated Verification 
(GV) pilot and comparison districts only. Differences between participation rates in pilot and comparison 
districts should not be interpreted as estimates of the impact of GV on NSLP participation rates. See 
Table III.7 for impact estimates. 

“Standard errors are in parentheses. These standard errors have been adjusted to account for the complex sampling 
design of the data set. 
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(that is, with incomes of no more than 185 percent of poverty), excluding those directly certified, 
is similar (89.8 percent). However, among the higher- income students not eligible for free or 
reduced-price meals, the reported overall participation rate among non-directly certified students 
was 73.6 percent, with most higher-income participants getting paid lunches. 

As we did in UFD districts, we examined the proportion of meals served to students in 
different income groups in GV districts by reimbursement status (Table III.6b). Students who 
did not get a school lunch were excluded from these calculations. Among non-directly certified 
students income-eligible for free meals who got a school meal, 70.0 percent got free meals, 11.7 
got reduced-price meals, and 18.4 percent got paid meals (Table III.6b). Among all students 
from households below 130 percent FPL, including those directly certified, 79.8 percent got free 
meals, 7.6 percent got reduced-price meals, and 12.6 percent got paid meals. Thus, only about 
one of eight meals served to students eligible for free meals were full-price meals. Among meals 
served to students with incomes of no more than 185 percent of poverty (including those directly 
certified), 66.1 percent were free, 13.7 were reduced-price, and 20.2 percent were paid. A large 
majority of lunches served to students not income-eligible for free or reduced-price meals in GV 
comparison districts were paid (87.8 percent), with 8.7 percent reduced-price and 3.5 percent 
free. 

Rates of NSLP participation in GV pilot districts may have been different from those in GV 
comparison districts, due either to the effects of the pilot intervention or to differences between 
the two types of districts in student/household characteristics or both. In general, the proportion 
of non-directly certified students in GV pilot districts reported to be receiving free (or reduced- 
price) meals was lower than in comparison districts. For example, the free meal participation 
rate among non-directly certified students was 20.8 percent in pilot districts, compared with 24.2 
percent in comparison districts (Table III.6a). 
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TABLE III.6b 



REIMBURSEMENT CATEGORY OE MEALS RECEIVED BY NSLP PARTICIPANTS IN GRADUATED 
VERIEICATION PILOT AND COMPARISON DISTRICTS, PALL 2002 



Percentage Distribution of 
Participants by Reimbursement 
Category’’ 

Comparison 

Student’s Estimated Eligibility Status’* Pilot Districts Districts 



Estimates Excluding Directly Certified Students 

Students Eligible for Pree Meals (Income <130% PPL) 



Pree lunch (P6 / P5) 


61.0 


70.0 




(4.67) 


(4.10) 


Reduced-price lunch (P7 / P5) 


7.5 


11.7 




(2.55) 


(1.60) 


Paid lunch (P8/P5) 


31.6 


18.4 




(4.77) 


(4.20) 


Students Eligible for Pree or Reduced-Price Meals (Income <185% PPL) 


Pree lunch (P10/P9) 


48.9 


56.3 




(3.26) 


(2.83) 


Reduced-price lunch (PI 1 / P9) 


15.7 


17.7 




(2.52) 


(1.48) 


Paid lunch (P12/P9) 


35.3 


25.9 




(3.62) 


(3.04) 


Students Not Eligible for Pree/Reduced-Price Meals (Income >185% PPL) 


Pree lunch (P14/P13) 


2.2 


3.5 




(0.91) 


(0.88) 


Reduced-price lunch (PI 5 / PI 3) 


3.7 


8.7 




(0.51) 


(2.77) 


Paid lunch (P16/P13) 


94.1 


87.8 




(1.02) 


(2.81) 


Estimates for All Students, Including Those Directly Certified 

Students Eligible for Pree Meals (Income <130% PPL) 


Pree lunch (P6 / P5) 


75.9 


79.8 


Reduced-price lunch (P7 / P5) 


4.8 


7.6 


Paid lunch (P8/P5) 


19.3 


12.6 


Students Eligible for Pree or Reduced-Price Meals (Income <185% PPL) 


Pree lunch (P10/P9) 


62.5 


66.1 


Reduced-price lunch (PI 1 / P9) 


11.6 


13.7 


Paid lunch (P12/P9) 


25.9 


20.2 



PPL = federal poverty level. 

“NSLP participation is reported by parents in the study survey. 

’’Standard errors are in parentheses. These standard errors have been adjusted to account for the complex sampling 
design of the data set. 
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On the other hand, the paid participation rate among non-directly certified students in pilot 
districts (49.4 percent) was a bit higher than in comparison districts (47.9 percent). On balance, 
the free/reduced-price meal difference outweighed the paid meal difference, so the overall 
participation rate in pilot districts was a bit lower than in comparison districts (for example, 77.9 
versus 81.5 percent among all non-directly certified students). These patterns held both overall 
and for each of the three income subgroups we examined. The patterns are also consistent with 
the reduction in free and reduced-price meals and increase in paid meals with small net change in 
total meals that the Food and Nutrition Service (2002) reported during the first year of pilot 
operation compared to the two prepilot years. 

2. Impacts on Participation Rates 

The estimated impact of GV on the overall participation rate of non-directly certified 
students is not statistically significant. After controlling for student/household characteristics, 
the impact estimates indicated that participation in pilot districts was lower than in comparison 
districts both overall and for free or reduced-price meal participation — by 3 to 5 percentage 
points — but these estimates were not significant at the 95 percent level (Table III.7). 

Among students income eligible for free meals, after controlling for student characteristics, 
the overall regression-adjusted participation rate was 5.4 percentage points lower in pilot districts 
than in comparison districts; this impact was not statistically significant at the 0.05 level. The 
reduction in the overall participation rate reflected a combination of a negative and statistically 
significant impact (of 13.3 percentage points) of the pilot on free/reduced-price participation 
among non-directly certified low-income students, combined with a positive 8.0 percentage point 
effect on paid participation that was not statistically significant at the 0.05 level. When free/ 
reduced-price income-eligible students were examined, all the estimated effects were smaller, 
and none was statistically significant. 
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TABLE III.7 



ESTIMATED IMPACT OE GRADUATED VERIEICATION PILOT PROJECTS ON NSLP PARTICIPATION 
(AS REPORTED BY PARENTS) AMONG STUDENTS NOT DIRECTLY CERTIEIED 





Regression-Adjusted Rate“ 




Measure of Participation 


Pilot 


Comparison 


Impact’’ 


All Students 








Any school lunch (PI) 


78.0 


81.4 


-3.4 

(2.82) 


Tree lunch (P2) 


21.1 


24.1 


-3.0 

(2.36) 


Tree or reduced-price lunch (P2 h-P3) 


28.6 


33.3 


- 4.1 

(2.65) 


Paid lunch (P4) 


49.2 


48.0 


1.2 

(3.65) 


Students Eligible for Eree Meals (Income 
<130% EPL) 








Any school lunch (P5) 


85.6 


91.1 


-5.4 

(3.38) 


Eree lunch (P6) 


52.7 


63.8 


-11.1 

(5.90) 


Eree or reduced-price lunch (P6 h-P7) 


60.2 


73.5 


-13.3* 

(6.00) 


Paid lunch (P8) 


25.4 


17.4 


8.0 

(5.90) 


Students Eligible for Eree or Reduced-Price 
Meals (Income <185% PPL) 








Any school lunch (P9) 


87.3 


88.7 


-1.3 

(3.04) 


Eree lunch (PIO) 


44.6 


48.5 


-3.9 

(4.93) 


Eree or reduced-price lunch (PlO-tPl 1) 


57.9 


64.7 


-6.8 

(5.11) 


Paid lunch (P12) 


29.1 


23.9 


5.1 

(4.94) 


Students Not Eligible for Pree/Reduced-Price 
Meals (Income >185% PPL) 








Any school lunch (PI 3) 


69.3 


74.9 


-5.6 

(4.03) 


Eree lunch (PI 4) 


1.8 


2.4 


-0.6 

(0.96) 


Eree or reduced-price lunch (P14H-P15) 


5.4 


7.9 


-2.6 

(1.68) 


Paid lunch (PI 6) 


63.8 


67.0 


-3.1 

(4.31) 



EPL = federal poverty level. 
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TABLE III. 7 (continued) 



“The regression-adjusted rate shows the estimated participation rate that would result under the assumption that 
students in both GV pilot and comparison districts had the same distribution of observable characteristics. Thus, 
any remaining difference between the regression-adjusted participation rate in pilot versus comparison district can 
be attributed to the impact of the pilot intervention. 

’’Standard errors are in parentheses. These standard errors have been adjusted to account for the complex sampling 
design of the data set. 

*Significantly different from zero at the .05 level, two-tailed test. 

**Significantly different from zero at the .01 level, two-tailed test. 
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Among students not income eligible for free or reduced-price meals, GV was estimated to 
have a negative effect on the overall participation rate of 5.6 percentage points, but this estimate 
was not statistically significant (Table III.7). Nor were the estimated effects of GV on the free, 
free/reduced-price, or paid meal participation rates statistically significant. 

Overall, the impact estimates suggest that GV reduced free or reduced-price participation 
among students income eligible for free meals — that is, the intervention may have discouraged 
low-income students from getting free or reduced-price meals. This discouraging effect 
probably occurred because of the impact of GV on the rate of free/reduced-price certification 
among this group. Burghardt et al. (2004) found that GV raised barriers to certification among 
students income eligible for free meals. With fewer students certified, fewer were getting free or 
reduced-price meals. However, many of the students discouraged from being certified (and thus 
no longer getting free or reduced-price meals) may have gotten school lunches at full price. In 
particular, we observed an increase in the paid participation rate among students income eligible 
for free meals. Although this estimated positive impact was not statistically significant, it was 
relatively large (8 percentage points, or 46 percent of the paid participation rate among free meal 
income-eligible students in GV comparison districts). Among groups other than those income 
eligible for free meals, however, GV did not have a statistically significant effect on 
participation; thus, the estimated effects of GV on overall participation rates among all students 
were not statistically significant either. 

3. Descriptive Data on Levels of Participation Accuracy 

As in UFD districts, we examined participation accuracy in GV districts using four different 
measures — one measure of free meal participation accuracy and three measures of free/reduced- 
price meal accuracy — for both non-directly certified students and for all students. The free meal 
accuracy rate excluding directly certified students in GV comparison districts was 73.3 percent in 
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fall 2002 (Table III.8). In other words, just under three-fourths of free meals served to non- 



directly certified students in GV comparison districts were served to students income eligible for 
these benefits — that is, those with incomes of no more than 130 percent of poverty. Among all 
students served free meals (including those directly certified), the free meal accuracy rate was 
82.2 percent. The estimated free/reduced-price meal accuracy rate depended on the definition of 
accuracy used. According to the most stringent definition of accuracy, 65.0 percent of free or 
reduced-price meals served to non-directly certified students went to those income eligible for 
exactly the benefit level they received. The other estimates indicated that 72.4 percent of 
free/reduced-price meals were served to students who were not overcertified, and 85.0 percent 
were served to students income eligible for either free or reduced-price meals. 

Rates of participation accuracy were very similar in GV pilot districts and comparison 
districts — in both cases, about 73 percent of free meals obtained by non-directly certified 
students were estimated to be served to income-eligible students. Free/reduced-price 
participation accuracy rates tended to be somewhat higher in pilot districts than in comparison 
ones. The three free/reduced-price participation accuracy rates among non-directly certified 
students ranged from 70.2 to 91.4 percent in GV pilot districts, compared with 65.0 to 85.0 
percent in comparison districts (Table III.8). The accuracy rates were higher in GV pilot districts 



1 

An additional 19.0 percent of free meals served to non-directly certified students in GV 
comparison districts were served to students who were income eligible for reduced-price 
meals — that is, those with incomes of between 130 and 185 percent of poverty. In other words, 
an estimated 92.3 percent of free meals served to non-directly certified students in GV 
comparison districts were served to students who were income eligible for either free or reduced- 
price meals. 
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TABLE III.8 



PARTICIPATION ACCURACY AMONG STUDENTS IN GRADUATED VERIEICATION 
PILOT AND COMPARISON DISTRICTS, PALL 2002 





Estimated Rate’’ 


Participation Accuracy Measures” 


Pilot Districts 


Comparison 

Districts 


Estimates Excluding Directly Certified Students 






Tree Meal Accuracy 

PA_E: Percentage of Tree Meals Served to Students Income Eligible 
for Eree Meals” 


73.0 

(4.56) 


73.3 

(3.38) 


Eree/Reduced-Price Meal Accuracy 

PA_ERP1; Percentage of Eree or Reduced-Price Meals Served to 
Students Eligible for the Level of Benefits They Are Receiving 


70.2 

(3.82) 


65.0 

(3.51) 


PA_ERP2; Percentage of Eree or Reduced-Price Meals Served to 
Students Not Overcertified for Benefits They Are Receiving 


76.7 

(3.40) 


72.4 

(3.42) 


PA_ERP3: Percentage of Eree or Reduced-Price Meals Served to 
Students with Incomes Below 185% of the PPL 


91.4 

(1.98) 


85.0 

(3.08) 


Estimates for All Students, Including Those Directly Certified 






Eree Meal Accuracy 

PA_P: Percentage of Eree Meals Served to Students Income Eligible 
for Eree Meals 


85.6 


82.2 


Pree/Reduced-Price Meal Accuracy 

PA_PRP1: Percentage of Eree or Reduced-Price Meals Served to 
Students Eligible for the Level of Benefits They Are Receiving 


80.8 


73.9 


PA_PRP2; Percentage of Tree or Reduced-Price Meals Served to 
Students Not Overcertified Benefits They Are Receiving 


85.2 


79.4 


PA_PRP3; Percentage of Tree or Reduced-Price Meals Served to 
Students with Incomes Below 185% of the PPL 


94.8 


88.8 



PPL = federal poverty level. 



Note; This table is intended to provide descriptive information on participation accuracy rates in Graduated 
Verification (GV) pilot and comparison districts only. Differences between accuracy rates in pilot and 
comparison districts should not be interpreted as estimates of the impact of GV on NSLP participation 
accuracy. See Table III. 9 for impact estimates. 

“NSLP participation is reported by parents in the study survey. 

’’Standard errors are in parentheses. These standard errors have been adjusted to account for the complex sampling 
design of the data set. 

“Among students who are served free meals, 94.8 percent of those in pilot districts and 92.3 percent of those in 
comparison districts are income eligible for either free or reduced-price meals. 
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than in GV comparison districts because partieipation rates of ineome-ineligible students were 
lower in pilot distriets than in eomparison ones.'"^ It is important to keep in mind that these 
aeeuracy rates were ealeulated without eontrolling for any student/household eharacteristies and 
therefore pilot-eomparison differenees in the aeeuraey rate do not imply that the pilot neeessarily 
eaused the difference. The following seetion presents estimated impacts of GV on participation 
accuracy. 

4. Impacts on Participation Accuracy 

Table III.9 shows the regression-adjusted partieipation aeeuraey rates among non-direetly 
eertified students in GV pilot and eomparison distriets, along with the estimated impaet of the 
pilot on participation accuracy. For each measure of partieipation aeeuraey, the estimated impact 
of GV was not statistically significant. In each case, the regression-adjusted partieipation 
aeeuracy rate in pilot distriets was greater than the regression-adjusted participation accuracy rate 
in eomparison distriets. However, the estimated standard errors of these impaet estimates were 
so large that the observed pilot-eomparison difference eould be due to sampling variability. 
Furthermore, as with UFD, the estimated impaet of GV on free/redueed-priee partieipation 
aeeuraey was sensitive to the definition we used to measure free/reduced-price aeeuraey. 

5. Reimbursement Accuracy 

In GV comparison districts, about I of 10 dollars (or 10.7 percent) of federal 
reimbursements for free, reduced-price, or paid meals (excluding directly certified meals) was 

In contrast, participation rates among eligible students were also lower in pilot districts 
than in comparison ones. In isolation, this difference would have led to lower participation 
accuracy in the pilot districts. However, this difference among eligible students was more than 
offset by the fact that students ineligible for free (or free/reduced-price) were less likely to 
participate in pilot districts, leading to greater accuracy in those districts. 
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TABLE III.9 



ESTIMATED IMPACT OE GRADUATED VERIEICATION ON PARTICIPATION ACCURACY 
AMONG STUDENTS NOT DIRECTLY CERTIEIED, EALL 2002 





Regression- 


-Adjusted Rate’’ 




Participation Accuracy Measures” 


Pilot 


Comparison 


ImpacL 


Tree Meal Accuracy 








PA_E: Percentage of Tree Meals Served to 
Students Income Eligible for Eree Meals 


74.9 


73.1 


1.71 

(10.77) 


Eree/Reduced-Price Meal Accuracy 








PA_ERP 1 ; Percentage of Eree or Reduced-Price 
Meals Served to Students Eligible for the Level 
of Benefits They Are Receiving 


69.6 


65.5 


4.11 

(8.71) 


PA_ERP2: Percentage of Eree or Reduced-Price 
Meals Served to Students Not Overcertified for 
the Level of Benefits They Are Receiving 


76.3 


73.0 


3.28 

(9.07) 


PA_ERP3: Percentage of Eree or Reduced-Price 
Meals Served to Students with Incomes Below 
185% of the PPL 


95.3 


83.1 


12.2 

(8.82) 



EPL = federal poverty level. 

“NSLP participation is reported by parents in the study survey. 

’’The regression-adjusted rate shows the estimated participation accuracy rate that would result under the assumption 
that students in both GV pilot and comparison districts had the same distribution of observable characteristics. 
Thus, any remaining difference between the regression-adjusted accuracy rate in pilot versus comparison district 
can be attributed to the impact of the pilot intervention. 

‘’Standard errors are in parentheses. These standard errors have been adjusted to account for the complex sampling 
design of the data set. 

*Significantly different from zero at the .05 level, two-tailed test. 

**Significantly different from zero at the .01 level, two-tailed test. 
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paid to districts over and above what they would have been paid if no students were overcertified 
(Table III. 10). The rate of reimbursements in excess of students’ income eligibility status in GV 
pilot districts was lower, at 6.9 percent. As in UFD districts, the amount by which 
reimbursements fall below what should have been paid to districts if all certified students had 
received their maximum level of benefits — that is, if all certified students with incomes below 
130 percent of poverty had received free meals — was much smaller. If the eligibility status of all 
of these undercertified students was changed so that their benefits were increased to the 
appropriate level given their income, federal reimbursements in GV comparison districts would 
have increased by only 1.1 percent. The comparable rate in GV pilot districts was similar: 0.9 
percent. 



Again, this ignores another type of undercertification — students not certified for either 
free or reduced-price meals when they are income-eligible for these benefits. 
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TABLE III. 10 



FEDERAL REIMBURSEMENTS FOR LUNCHES RECEIVED BY STUDENTS WHO ARE NOT DIRECTLY 
CERTIFIED IN GRADUATED VERIFICATION PILOT AND COMPARISON DISTRICTS, FALL 2002 





Estimated Rate’’ 


Reimbursement Measures“ 


Pilot Districts 


Comparison 

Districts 


Percentage of all Federal Reimbursements for Meals Served to Students 
Not Eligible for the Level of Benefits They Are Receiving 


6.9 

(1.2) 


10.7 

(1.8) 


Percentage Short-Fall in Federal Reimbursements Because Some Certified 
Students Received a Lower Benefit Than They Were Eligible for 


0.9 

(0.3) 


1.1 

(0.2) 



Note: This table is intended to provide descriptive information on reimbursement accuracy in Graduated 

Verification (GV) pilot and comparison districts only. Differences between rates in pilot and comparison 
districts should not be interpreted as estimates of the impact of GV on reimbursement accuracy. 

“Reimbursements are calculated by using student participation as reported by parents in the study survey and 
applying the reimbursement appropriate for the student’s meal price status. 

’’Standard errors are in parentheses. These standard errors have been adjusted to account for the complex sampling 
design of the data set. 
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IV. SUMMARY OF IMPACTS 



This report has examined the effects of Up-Front Documentation (UFD) and Graduated 
Verification (GV) on participation in the NSLP and is part of a larger study examining the 
overall effects of these interventions on the NSLP. The study has focused largely on outcomes 
related to (1) deterrence of students who are income ineligible for free or reduced-price meals 
from becoming certified for those benefits, and (2) barriers to free/reduced-price meals 
certification among students who are income eligible for those benefits. The pilot interventions 
were designed to deter income-ineligible students from becoming certified, and it was hoped that 
this could be accomplished without raising barriers to certification among income-eligible 
students. Burghardt et al. (2004) addressed the question of what impacts UFD and GV had on 
rates of certification among eligible and ineligible students. 

Simply examining how the pilot interventions affected certification is incomplete, however. 
The main reason students become certified is to get school meals free or at a reduced price. 
Thus, whether students become certified presumably affects whether they get school lunches. 
By extension, if the pilot interventions influence certification rates, then they may also influence 
rates of participation. Furthermore, knowing whether any impacts on certification translate into 
impacts on participation helps us interpret the impacts on certification. If UFD and GV lead to 
lower certification rates but rates of free/reduced-price, paid, and overall participation are 
unchanged, then the impacts on certification rates become less important. If, on the other hand, 
the lower certification rates in pilot districts translate into lower participation rates, then this 
could have implications for children’s dietary intake, families’ spending on their children’s 
meals, the extent to which the NSLP provides income support to low-income families in the 
manner the National School Lunch Act intended, and SFAs’ budgets. 
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This chapter summarizes the overall impacts of UFD and GV on rates of school lunch 
participation. The chapter also places these participation impact results into context by 
comparing them to estimated impacts on certification as reported in Burghardt et al. (2004). 
Section A summarizes the impacts of UFD, while Section B provides analogous information on 
the impacts of GV. 

A. UP-FRONT DOCUMENTATION 

Overall, UFD reduced the proportion of all students getting a school lunch on a given day. 
The estimated impact of the pilot on the overall participation rate (as reported by parents) was 
-5.8 percentage points and was statistically significant (Table IV. 1). This impact represents a 
reduction in participation of 8.0 percent from the rate observed in comparison districts. The 
overall reduction in participation came from a combination of decreases in the proportion of 
students getting free/reduce-price meals and the proportion getting paid meals. In particular, the 
UFD was estimated to reduce the free/reduced-price participation rate by 15.1 percent (which 
was statistically significant) and to reduce the paid participation rate by 6.1 percent (which was 
not statistically significant). 

These estimated impacts of UFD on rates of participation among all students are reasonably 
consistent with observed changes in participation rates during the first year after the 
implementation of the pilot, as presented in the USDA descriptive study of the NSLP 
Application/Verification Pilot Projects (U.S. Department of Agriculture 2002). Based on the 
USDA study, the observed decline in the number of free/reduced-price meals served per day in 
eight UFD districts in the first year after the pilot was 16 percent, close to the negative 15 
percent impact described above. Both studies also show a negative relationship between UFD 
and the overall level of NSLP participation. On the other hand, while this study found a negative 
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TABLE IV. 1 



SUMMARY OF ESTIMATED IMPACTS OF UP-FRONT DOCUMENTATION 
AMONG NON-DIRECTLY CERTIFIED STUDENTS 



Measure of Participation/Certification 


Mean Rate in 
Comparison 
District 


Estimated 
Percentage Points 


1 Impact 

As Percent of 
Comparison 
District Mean 


All Students 


Free/reduced-price participation rate 


18.5 


-2.8* 


-15.1 


Paid participation rate 


54.0 


-3.3 


-6.1 


Overall participation rate 


72.6 


-5.8** 


-8.0 


Students Eligible for Free Meals (Income 
<130% FPL) 


Free/reduced-price certification rate^ 


53.7 


-6.2 


-11.5 


Free/reduced-price participation rate 


57.4 


-7.8 


-13.6 


Paid participation rate 


26.1 


1.8 


6.9 


Overall participation rate 


83.5 


-5.4 


-6.5 


Students Eligible for Free or Reduced-Price 
Meals (Income <185% FPL) 


Free/reduced-price certification rate^ 


51.1 


-9.1* 


-17.8 


Free/reduced-price participation rate 


46.1 


_9 3** 


-20.2 


Paid participation rate 


36.3 


-1.4 


-3.9 


Overall participation rate 


82.5 


-9.8** 


-11.9 


Students Ineligible for Free or Reduced-Price 
Meals (Income >185% FPL) 


Free/reduced-price certification rate^ 


3.9 


0.0 


0.0 


Free/reduced-price participation rate 


3.5 


-0.2 


-5.7 


Paid participation rate 


64.1 


-4.6 


-7.2 


Overall participation rate 


67.6 


-4.4 


-6.5 



“The certification rate estimates are taken from Burghardt et al. (2003). 
FPL = federal poverty level. 

*Significantly different from zero at the .05 level, two-tailed test. 
**Significantly different from zero at the .01 level, two-tailed test. 
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impact of UFD on paid participation, the U.S. Department of Agriculture (2002) found that the 
observed level of paid participation increased in the first year after the implementation of the 
pilot/ 

To fully understand how UFD affected the participation behavior of students in pilot 
districts, however, we need to know how the pilot influenced participation rates of two different 
groups of students: (1) those income eligible for free/reduced-price meals, and (2) those not 
income eligible for these benefits. This distinction is important because UFD was designed with 
the hope that it would have differential effects on these two groups of students. By requiring 
students applying for benefits to submit documentation of the information provided on their 
application, the hope was that income-ineligible students would be deterred from becoming 
certified without discouraging those who were income eligible for benefits from becoming 
certified. Presumably, these differential effects on certification between the two groups would 
also lead to differential effects on participation. So while it is interesting to know that our 
estimates suggest that UFD led to a decline in both free/reduced-price and paid participation, we 
are particularly interested in knowing whether these effects primarily occurred among those low- 



* There are several reasons why the estimated impacts presented in this report could differ 
from the observed changes presented in the year one descriptive report (U.S. Department of 
Agriculture 2002). Most important, the study findings presented here are based on a design 
intended to estimate the impacts of UFD and GV. Thus, we have attempted to control for factors 
potentially affecting participation but not related to the pilot intervention itself. The descriptive 
report, on the other hand, did not attempt to control for such factors; its objective was to describe 
observed changes in outcomes after the implementation of the pilot. Three other differences are 
worth mentioning. First, this report examines outcomes from the 2002-2003 school year, two 
years after the year one descriptive report. Second, this report includes information from nine 
UFD pilot districts, including two that first implemented UFD in the 2001-2002 school year and 
not including one included in the USDA first-year analysis. Third, this report excluded directly 
certified students from the sample, while the figures presented in the descriptive report included 
all students. 
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income students eligible for benefits, those higher-income students not eligible for benefits, or 
some combination of both groups. 

Among students income eligible for free/reduced-price meals, UFD was estimated to lead to 
an 11.9 percent reduction in NSLP participation (Table IV. 1). This reduction in the overall 
participation rate arose essentially from a reduction of 20.2 percent in the rate of free/reduced- 
price participation. Both of these estimated impacts were statistically significant. The estimated 
impact of UFD on free/reduced-price participation closely matched the estimated impact on the 
free/reduced-price certification rate among students from households income eligible for free or 
reduced-price meals, presented in Burghardt et al. (2004), which was -17.8 percent (excluding 
those directly certified). Among income-eligible students, the estimated impact on paid 
participation was small and not statistically significant. 

Thus, among students from households income eligible for free or reduced-price meals, 
UFD led to a decline in both free/reduced-price certification and participation, with no effect on 
paid participation. One explanation consistent with this pattern of impacts is that the pilot 
intervention discouraged some students from becoming certified for benefits, and these students 
stopped getting school lunches as a result. The fact that the free/reduced-price participation rate 
declined as a result of the intervention — and that this decline matched the decline in 
free/reduced-price certification — suggests that the students discouraged from becoming certified 
would have gotten free or reduced-price school lunches if UFD had not been implemented. In 
addition, the fact that there was essentially no change in paid participation as a result of UFD 
suggests that this group of students discouraged from becoming certified did not continue to get 
school lunches at the full price. If they had, we would likely have observed an increase in the 
paid participation rate. 



75 




Among students who were income ineligible for free/reduced-price meals, the estimated 
effects of UFD on certification and participation were smaller and not statistically significant. 
Just as Burghardt et al. (2004) found no impact on the free/reduced-price certification rate among 
this group, our estimate of the impact on the free/reduced-price participation rate is small and not 
statistically significant. However, UFD may have led to a modest decrease in the paid 
participation rate and a corresponding modest decrease in the overall participation rate among 
students not income eligible for free/reduced-price meals. Both of these estimated impacts had 
relative magnitudes of about -7 percent, although they were not significant at the 0.05 level of 
statistical significance. Despite not being significant at the 0.05 level, we believe that these 
negative impacts on paid and overall participation represented actual impacts of UFD and did not 
arise solely from chance, for two reasons. First, the estimates would have achieved statistical 
significance if we had used the 0.10 level as the standard of significance. Second, the estimated 
negative impacts were supported by evidence from an analysis of administrative data on NSLP 
participation that UFD reduced the percentage of students receiving paid meals (Appendix Table 
A.4). 

If UFD actually caused reductions in paid participation among students not income eligible 
for free/reduced-price meals, this finding is unexpected because the UFD pilot was designed to 
directly affect only those who applied (or considered applying) for free/reduced-price meal 
benefits. If anything, we would have expected to see an increase in paid participation among 
income-ineligible students, as some of those who were deterred from becoming certified 
continued to get NSLP lunches at the paid rate. 

One possible explanation for this unexpected negative impact is the “peer” effect. If the 
participation behavior of other students influences a given student’s decision to get a school 
lunch, it is possible that the negative impact of UFD on free/reduced-price participation among 
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students who were income eligible for those benefits spills over into a negative impact on paid 
participation among students not income eligible for free/reduced-price meals. In other words, 
students not directly affected by UFD may stop getting paid meals because their friends, who 
were directly affected by UFD, have stopped getting free/reduced-price meals. 

UFD did not affect participation accuracy to an extent that we could detect with the study 
sample and measures. Point estimates of the effects varied widely depending on the specific 
measure used, and they were sensitive to the inclusion or exclusion of one pilot district that 
exhibited unusual characteristics (an unusually small fraction of students certified for free and 
reduced-price meals, and small numbers of students reported by parents to get free or reduced- 
price meals). However, the point estimates suggest that the percentage of benefits going to 
income-eligible students was not increased and may even have been reduced slightly. 
Furthermore, a small reduction in participation accuracy is consistent with the finding that free 
and reduced-price participation among income-eligible students was reduced, while free and 
reduced-price participation among income-ineligible students was unchanged. 

B. GRADUATED VERIFICATION 

The estimated impacts of GV on participation rates among all students tended to be small 
and were not statistically significant. Overall, based on parental reports, GV was estimated to 
lead to a small reduction in the proportion of all students getting a school lunch on a given day in 
fall 2002: -3.4 percentage points (Table IV. 2), which represents a reduction in participation of 
about 4.2 percent from the rate observed in comparison districts. The overall reduction in 
participation came from a combination of a 14 percent decrease in the proportion of students 
getting free/reduced-price meals and a 2.5 percent increase in the proportion getting paid meals. 
Again, none of these estimates was statistically significant at the 0.05 level, although the 
estimated impact on free/reduced-price participation was marginally significant (that is. 
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TABLE IV.2 



SUMMARY OF ESTIMATED IMPACTS OF GRADUATED VERIFICATION 



Measure of Participation/Certification 


Mean Rate in 
Comparison 
District 


Estimated Impact 

As Percent of 
Comparison 

Percentage Points District Mean 


All Students 


Free/reduced-price participation rate 


33.5 


-4.7 


-14.0 


Paid participation rate 


47.9 


1.2 


2.5 


Overall participation rate 


81.5 


-3.4 


-4.2 


Students Eligible for Free Meals (Income 
<130% FPL) 


Free/reduced-price certification rate^ 


81.1 


-15.6* 


-19.2 


Free/reduced-price participation rate 


75.2 


-13.3* 


-17.7 


Paid participation rate 


16.9 


8.0 


47.3 


Overall participation rate 


92.2 


-5.4 


-5.9 


Students Eligible for Free or Reduced-Price 
Meals (Income <185% FPL) 


Free/reduced-price certification rate^ 


71.1 


-9.1 


-12.8 


Free/reduced-price participation rate 


66.4 


-6.8 


-10.2 


Paid participation rate 


23.3 


5.1 


21.9 


Overall participation rate 


89.8 


-1.3 


-1.4 


Students Ineligible for Free or Reduced-Price 
Meals (Income >185% FPL) 


Free/reduced-price certification rate^ 


8.6 


-2.5 


-29.1 


Free/reduced-price participation rate 


9.0 


-2.6 


-28.9 


Paid participation rate 


64.6 


-3.1 


-4.8 


Overall participation rate 


70.8 


-5.6 


-7.9 



“The certification rate estimates are taken from Burghardt et al. (2003). 
FPL = federal poverty level. 

*SignificantIy different from zero at the .05 level, two-tailed test. 
**Significantly different from zero at the .01 level, two-tailed test. 
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significant at the 0.10 level). Since these estimates were based on data collected in the fall of 
2002, they do not reflect any effects of additional verification activities conducted in the pilot 
districts during the 2002-2003 school year as a result of the GV intervention. Instead, the 
estimated impacts capture any cumulative effects of the intervention as it was implemented in 
previous years. 

As with UFD, the estimated impacts of GV on the rate of free/reduced-price participation 
among all students were reasonably consistent with observed changes in this participation rates 
during the first year after the implementation of the pilot, as presented in the USDA descriptive 
study of the NSLP ApplicationA/^erification Pilot Projects (U.S. Department of Agriculture 
2002). That study reported a decline of 14 percent in the number of free/reduced-price meals 
served per day in the three GV districts included in the evaluation in the first year after the pilot 
was implemented, which matches the 14.0 percent impact described above. Both studies also 
showed a positive relationship between GV and the paid participation rate, although the observed 
year one changes presented in the U.S. Department of Agriculture (2002) study were larger than 
the estimated impact of GV on this outcome presented in Table IV. 2. Consequently, the 
observed year-one change in the overall participation rate, although negative, was smaller in 
magnitude than the impact presented in Table IV. 2. 

Table IV. 2 presents estimates of the impact of GV on participation rates separately for 
students who were income eligible for free or reduced-price meal benefits and for those not 
income eligible for these benefits. In addition, the table also presents impacts separately for the 
group that was income eligible for free meal benefits alone. 

Among students who were income eligible for free meals, GV led to a 17.7 percent 
reduction in the rate of free/reduced-price participation (Table IV. 2). This reduction in the 
free/reduced-price participation rate was about the same as the estimated impact of -19.2 percent 
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on the free/reduced-price certification rate among students who were income eligible for free 
meals presented in Burghardt et al. (2004). Both of these estimates were statistically significant. 
In contrast, GV led to an increase in the paid participation rate among income-eligible students, 
with the estimated rate in pilot districts being 8.0 percentage points, or 47.3 percent, higher in 
GV pilot districts than in comparison districts. However, this estimate was not statistically 
significant at the 0.05 level. This combination of a negative impact on free/reduced-price 
participation and a positive impact on paid participation resulted in a small (5.9 percent) — and 

not statistically significant — negative impact of GV on the overall NSLP participation rate 

2 

among students who were income eligible for free meals. 

As described in Burghardt et al. (2004), the estimated negative impact of GV on certification 
among this group suggests that some students who were income eligible for free meal benefits 
were discouraged from becoming certified for benefits as a result of GV. The fact that the 
estimated impacts on free/reduced-price participation were of a similar magnitude as the impacts 
on certification suggests that those income-eligible students discouraged from certification would 
have gotten free/reduced-price meals if GV had not been implemented. Thus, by reducing the 
number of income-eligible students certified for benefits, GV also reduced the number getting 
free/reduced-price meals. However, the pattern of impacts described above suggests that some 
(although not all) of those who were no longer getting free/reduced-price meals due to GV 
continued to get school lunches, although at the full-price, or paid, rate. This would explain a 



2 

The estimated impact of GV on participation among students eligible for free or reduced- 
price meals followed a pattern similar to that of the estimated impacts among those eligible for 
free meals. The magnitudes of these impacts were smaller — although still reasonably large in 
percentage terms — and were not statistically significant. 
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positive effect of GV on the paid participation rate and the fact that the estimated negative 
impact on the overall participation rate was relatively small. 

Among students who were income ineligible for free/reduced-price meals, the percentage 
getting free or reduced-price meals in comparison districts was not trivial. On a typical day, for 
example, 9.0 percent of these income-ineligible students in comparison district got free/reduced- 
price meals. GV was estimated to have a negative impact on this participation rate among 
income-ineligible students — the free/reduced-price participation rate was 2.6 percentage points 
(29 percent) lower in GV pilot districts than in comparison ones. The magnitude of this 
estimated impact was consistent with the magnitude of the estimated impact of GV on the 
free/reduced-price certification rate among income-ineligible students (Burghardt et al. 2004). In 
each case, however, this estimate was not statistically significant at the 0.05 level. These data 
suggest that GV lowered the rate of both free/reduced-price certification and participation among 
some students who were income-ineligible for these benefits, but we cannot rule out the 
possibility that these patterns are due to chance variation in our study sample. 

The estimated impact of GV on paid participation among students who were income 
ineligible for free/reduced-price benefits was also not statistically significant and was small in 
percentage terms, with the paid participation rate 3.1 percentage points (4.8 percent) lower in 
pilot districts than in comparison ones. Similarly, the estimated impact of GV on the overall 
NSLP participation rate among students who were income ineligible for free/reduced-price meals 
was small and negative, but not statistically significant. 

Overall, therefore, the strongest impacts of GV appear to be in reducing free/reduced-price 
participation among low-income students eligible for free meals. The pilot resulted in a 
substantial number of these students not becoming certified for free/reduced-price meals and, 
therefore, not getting these meals. However, some of the students affected in this way appear to 
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have continued to get school meals at the higher paid rate. Among students who were income 
ineligible for free/reduced price benefits, the pilot may have deterred some from become 
certified and getting free/reduced-price meals. However, this estimated impact was not 
statistically significant and did not greatly affect the overall participation rate among this group, 
since a large majority of the income-ineligible students who participate get paid meals. There 
was no hint that GV had a negative peer effect on the paid participation rate in fall 2002. 

Finally, GV did not affect participation accuracy to an extent that we could detect with the 
study sample and measures. The point estimates of the effects varied widely depending on the 
specific measure used, although each point estimate was consistent with a small increase in 
participation accuracy. In contrast to the situation for UFD districts, we cannot infer the 
direction of change in participation accuracy from the separate estimates of participation impacts 
for income-eligible students and income-ineligible students because the effects for the two 
groups were offsetting. For GV districts, we found a reduction in free and reduced-price 
participation of 6.8 percentage points among income-eligible students and a reduction of 2.6 
percentage points among income-ineligible students. The former will tend to decrease 
participation accuracy, while the latter will tend to increase it, and we cannot infer whether the 
net effect will be an increase or decrease. 
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APPENDIX A 

ANALYSIS OF ADMINISTRATIVE DATA 




As noted in Chapter II, the main data source for the analysis was survey data, on which 
parents reported the number of days per week their child usually eats a school lunch. A 
limitation of using this source of data to measure students’ NSLP participation is reporting error. 
Parents may not know whether their children are getting school lunches for which the local SFA 
receives federal reimbursement. In this Appendix, we use administrative data on aggregate 
levels of participation in pilot and comparison districts to address concerns about the accuracy of 
the parent-reported data. In participation, we address two questions: 

1 . Are participation levels reported by parents accurate (that is, consistent with the 
administrative data)? 

2. Are estimated impacts of the pilot demonstrations on participation (based on 
parent-reported data) accurate? 

A. DATA AND METHODS 
1. Administrative Data on NSLP Participation 

The Food and Nutrition Service (FNS) of USDA obtained administrative data on 
participation in the NSLP from each of the districts included in the evaluation. The data 
collected included: 

• The number of free, reduced-price, and paid meals served in the district in September 
through November 1999 and September through November 2002 

• The number of children enrolled and the number certified for free and reduced-price 
meals as of October 31, 1999 and October 31, 2002 

• The number of servings days per month in September-November 1999 and 
September-November 2002 

A copy of the form used to collect these data at Up-Front Documentation (alternative 1) pilot 
districts is shown as Exhibit A.l. Based on these data, FNS calculated average daily 
participation rates in September through November 1999 and 2002 for all students and for 
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subgroups of students certified for free meals, certified for free or reduced-price meals, and not 



certified. ' 

These participation rates are based on all students, including those directly certified for free 
meals. The participation rates presented in Chapter III of this report are based on all students 
except those who have been directly certified. Therefore, the next step in generating 
participation estimates based on administrative data comparable to those based on parent- 
reported survey data involved adjusting the participation rates based on administrative data to 
exclude directly certified students. A critical piece of information needed to do this is the 
number of directly certified students in each district. We collected data on the number of 
directly certified students in each district as part of the overall evaluation (see Burghardt et al. 
2004). We then assumed that participation rates among directly certified students are identical to 
those among other students within the same district who are certified for free meals (by 
application). This allowed us to adjust the overall participation rate in order to translate it into a 
participation rate among only non-directly certified students. 

2. Methods for Comparing Participation Rates from the Two Data Sources 

The first comparison we made between the two data sources involved whether the levels of 
NSLP participation were different when based on parents’ reports in the survey data versus when 
based on administrative data. Do parents accurately estimate, overestimate or underestimate the 
frequency with which their children eat school lunches? 



' The data collected by FNS and the methods used to measure participation rates based on 
the data are summarized in: Strasberg, Paul, “NSLP Application / Verification Pilot Projects: 
Analysis of the Effect on Participation,” internal memorandum, September 23, 2003. 
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EXHIBIT A. 1 



National School Lunch Program Application & Verification Pilot Projects 
School Food Authority Reporting Form for School Year 2001-02 
Alternative 1 

As of last operating 

Name of School Food Authority day in October 



1. SFA Enrollment (as of last operating day in October) 

Number of children with access to school lunch in schools operating the pilot this 
year 

2. Free and Reduced Price Applications (thru last operating day in October) 

A. Type of free and reduced price application (household or individual child) 

B. Total number of applications received (complete or incomplete) 

C. Number of applications received that lacked documentation of income 

D. Among applications in 2C., number that were re-submitted 

E. Among applications in 2C., number that were NOT re-submitted 

3. Free and Reduced Price Applications & Direct Certification 

A. Total of free approved children 

B. Free approved children included in A. approved by direct certification 

C. Free approved children included in A. approved by CATEGORICAL eligibility 



Note: CATEGORICAL eligibility is defined as a child whose parent/guardian 
completed an application which was approved by the school district for free meal 
benefits indicating current eligibility for Food Stamps, TANF or FDPIR benefits 



D. Free approved children included in A. approved on the basis of household 
size and income information provided on an application 

E. Reduced price approved children 

F. Completed applications for free/reduced price meal benefits that were 
DENIED based on income and family size 

G. Children represented in 3F on applications that were DENIED based on 
income and family size 



Please complete second page. 
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National School Lunch Program Application & Verification Pilot Projects 
School Food Authority Reporting Form for Schooi Year 2001-02 

Reimbursabie meais served and serving days, by month 

Name of School Food Authority 

Schooi Year 2001-02 

Lunches reimbursed 



August 

September 

October 

November 

December 

January 

February 

March 

April 

May 




Instructions: Complete and mail, fax or email this form to 



Dr. Paul J. Strasberg 

USDA Food and Nutrition Service 

3101 Park Center Drive 

Alexandria, VA 22302 

paul.strasberg@fns.usda.gov 

phone (703) 305 2141 ; fax (703) 305-2576 
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As described above, the administrative data measure free/reduced-price, paid, and total 
participation rates at the district level (adjusted to exclude directly certified students). To 
compare these rates to survey-based participation rates, we first had to aggregate the individual- 
level participation data by district to obtain survey-based estimates of district level participation 
rates. 

Next, we averaged both survey-based and administrative data-based participation rates 
across the nine Up-Front Documentation pilot districts and nine comparison districts, as well as 
across the three Graduated Verification pilot districts and four comparison districts. This 
allowed us to compare the overall participation rate estimates based on administrative versus 
survey data. A statistically significant difference in the estimated participation rates between the 
two data sources would suggest that the survey based information on students’ rates of 
participation is biased, assuming that the administrative data provides accurate information on 
NSLP participation. 

3. Methods for Comparing Estimated Impacts of the Pilots on Participation Rates 

Even if the estimated participation rates based on the two data sources differed, however, it 
will not necessarily imply that the estimated impacts of the pilots on participation rates will be 
biased. In other words, if parents overestimate their children’s frequency of participation, but do 
so to a similar extent in the pilot and comparison districts, our estimate of the effect of the pilot 
on participation would be unbiased. 



2 

Because two of the Graduated Verification comparison districts were for the same pilot 
district, we first computed the mean participation rate in these two districts and then computed 
the average of this mean across the two comparison districts to create a single comparison 
district participation rate for this pilot-comparison district pair. 
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In Chapter II of this report, we presented our methodology for estimating the impacts of the 
pilots on participation using survey data. This involved using regression analysis with 
individual-level survey data to compare probabilities of participation in pilot versus comparison 
districts, after controlling for relevant individual characteristics. Since we do not have 
individual-level administrative data on participation, we must use a different methodology to 
estimate impacts with this source of data. In particular, we estimated a fixed effects model using 
district-level administrative data on participation rates in pilot and comparison districts in both 
the fall of 2002 and in a similar period during a baseline year prior to the implementation of Up- 
Front Documentation and Graduated Verification. 



The fixed effects, or “difference-in-differences,” model can be written: 



pU2 _ p02 \ _ z' p9y _ pW \ 

pilot comparison * ^ pilot ^ comparison * 



The idea behind this estimate is that the first term examines how participation rates differ in pilot 
and comparison districts after the pilot has been in place. This difference would indicate how the 
pilot affected participation if all else were equal. But all else may not be equal. To control both 
for any pre-existing differences in pilot and comparison districts and for changes over time in 
conditions in pilot and comparison districts that could affect participation, we simply subtract 
the difference in baseline participation rates between pilot and comparison districts. This 
“difference-in-differences” is our estimate of the overall impact based on administrative data. 

We estimated difference-in-difference impact estimates separately for each pilot-comparison 
district pair. Then, we averaged the nine Up-Front Documentation estimates to come up with an 
overall estimate of the impact of Up-Front Documentation on participation, and averaged the 
four Graduated Verification estimates to obtain the estimate of the impact of Graduated 
Verification. 
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B. RESULTS 



1. Level of NSLP Participation 

Table A.l presents the mean estimated participation rates in 2002 for all districts within each 
of four groups — (1) Up-Front Documentation pilot districts, (2) Up-Front Documentation 
comparison districts, (3) Graduated Verification pilot districts, and (4) Graduated Verification 
comparison districts — based on both parents’ reports of their children’s participation and on the 
administrative data provided by FNS. This measure is the percentage of students getting a NSLP 
lunch on a typical school day (excluding directly certified students). In addition, the final column 
shows the average difference between the participation rate in pilot versus comparison districts. 
In addition to the overall participation rate, the table shows the free/reduced-price and paid 
participation rates according to the two data sources.^ 

The table indicates that parents reported higher levels of NSLP participation on the survey 
than are consistent with the administrative data on participation. According to the parent- 
reported data, 72.6 percent of students got a lunch in the average Up-Front Documentation 
comparison district in fall 2002. However, according to administrative data, the average 



-5 

Tables A. 2 and A. 3 present the estimated participation rates in each pilot and comparison 
district separately, which represent the data used to calculate the numbers shown in Table A.l. 
Table A. 2 presents participation rates estimates for each district based on administrative data for 
2002 and also for the baseline year of 1999. 



A.9 




TABLE A. 1 



ESTIMATED PARTICIPATION RATES IN 2002 BASED ON SURVEY AND ADMINISTRATIVE DATA IN 
UP-ERONT DOCUMENTATION AND GRADUATED VERIEICATION SITES, BY PILOT STATUS 



Measure of Participation 


Mean Participation Rate 
Pilot Comparison 


Difference 


Up-Front Documentation Sites 






Total Participation Rate 


Survey data 


66.1 


72.6 


-6.5 


Administrative data 


42.0 


48.1 


-6.0 


(Survey-based rate) - (Admin-based rate) 


24.1 


24.5 




Eree/Reduced-Price Participation Rate 


Survey data 


15.4 


18.5 


-3.1 


Administrative data 


13.4 


16.6 


-3.3 


(Survey-based rate) - (Admin-based rate) 


2.0 


1.9 




Paid Participation Rate 


Survey data 


50.5 


54.0 


-3.5 


Administrative data 


28.7 


31.4 


-2.7 


(Survey-based rate) - (Admin-based rate) 


21.8 


22.6 




Graduated Verification Sites 






Total Participation Rate 


Survey data 


77.9 


81.5 


-3.6 


Administrative data 


65.8 


62.3 


3.5 


(Survey-based rate) - (Admin-based rate) 


12.1 


19.2 




Eree/Reduced-Price Participation Rate 


Survey data 


28.1 


33.6 


-5.5 


Administrative data 


28.2 


30.7 


-2.5 


(Survey-based rate) - (Admin-based rate) 


-0.1 


2.9 




Paid Participation Rate 


Survey data 


49.4 


47.9 


1.5 


Administrative data 


37.6 


31.6 


6.0 


(Survey-based rate) - (Admin-based rate) 


11.8 


16.3 
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percentage was 48.1 percent."^ We calculated a rate of over-reporting participation by calculating 
the percentage of students who were not participants (according to administrative data) who were 
reported to be participants (according to parent-reported survey data). In Up-Front 

Documentation comparison districts, this over-reporting rate was 47 percent.^ Over-reporting of 
NSLP participation occurred primarily among students not certified for free or reduced-price 
meals. The paid meal participation rate — the percentage of all students who got a school lunch 
at the full price — was 54.0 percent in Up-Front Documentation comparison districts according to 
parents and 31.4 percent according to administrative data (Table A.l). By contrast, the 
free/reduced-price participation rate according to the two data sources was much more similar — 
18.5 percent according to parents and 16.6 percent according to administrative data. 

Parents over-reported their children’s participation at similar rates in Up-Front 
Documentation pilot districts as in comparison districts. As a result, the estimated pilot- 
comparison difference in participation was similar whether based on survey or administrative 
data. In particular, the estimated participation rate was 6.5 percentage points lower in pilot 
districts than in comparison districts according to parents and 6.0 percentage points lower in pilot 
districts according to administrative data. This is a preliminary indication that estimated impacts 



^ A careful examination of the data on Tables A. 2 and A. 3 identifies one site in which 
parents’ over-reporting of their children’s NSLP participation was particularly severe. In UFD 
District Pair 7, the overall participation rate was 68 to 81 percent according to parents and 12 to 
1 8 percent according to administrative data. Due to the severe over-reporting in this district, we 
conducted a sensitivity analysis to determine whether the estimates from the basic models 
presented in this report would have changed if this district were excluded. The results of the 
model in which estimated impacts on participation rates were not substantially affected by the 
exclusion of this site. The magnitude of the estimated impacts on participation accuracy did 
change when this site was excluded, though these estimated impacts were not statistically 
significant regardless of whether the site was included or excluded. 

^ In particular, we divided the difference between the parent-reported participation rate and 
administrative data-based participation rate (72.6 - 48.1) by the estimated percentage of all 
students who did not get a school lunch (100 - 48.1). 
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TABLE A.2 



ESTIMATED PARTICIPATION RATES BASED ON ADMINISTRATIVE DATA IN 
UP-ERONT DOCUMENTATION AND GRADUATED VERIEICATION SITES, 
BY DISTRICT AND PILOT STATUS 



Mean Participation Rate“ Difference 



2002 1999 



Measure of Participation 


Pilot 


Comparison 


Pilot 


Comparison 


2002 


1999 






Up-Front Documentation Sites 






Total Participation Rate 


District pair 1 


36.2 


51.7 


37.0 


48.6 


-15.2 


-11.6 


District pair 2 


61.8 


63.8 


61.8 


68.8 


-2.0 


-7.0 


District pair 3 


45.5 


32.2 


50.2 


23.2 


12.9 


27.0 


District pair 4 


38.4 


50.1 


40.2 


52.8 


-11.8 


-12.7 


District pair 5 


44.0 


47.0 


32.7 


43.9 


-3.0 


-11.2 


District pair 6 


66.3 


71.8 


72.4 


69.2 


-5.5 


3.2 


District pair 7 


12.4 


17.7 


15.5 


17.7 


-5.3 


-2.2 


District pair 8 


36.1 


42.7 


42.9 


47.8 


-6.6 


-4.8 


District pair 9 


38.0 


55.1 


38.3 


47.4 


-17.1 


-9.1 


Eree/Reduced-Price 
Participation Rate 


District pair 1 


9.7 


21.1 


10.4 


19.9 


-11.3 


-9.5 


District pair 2 


24.3 


19.7 


20.4 


23.0 


4.7 


-2.6 


District pair 3 


10.9 


12.4 


12.1 


10.6 


-1.5 


1.5 


District pair 4 


7.5 


12.3 


8.4 


12.3 


-4.8 


-3.9 


District pair 5 


12.9 


14.5 


10.3 


14.8 


-1.6 


-4.5 


District pair 6 


30.1 


23.6 


33.0 


24.8 


6.5 


8.2 


District pair 7 


5.1 


10.1 


8.7 


6.8 


-5.0 


2.0 


District pair 8 


4.2 


9.2 


5.8 


7.2 


-5.0 


-1.4 


District pair 9 


15.7 


27.1 


14.4 


25.6 


-11.3 


-11.2 


Paid Participation Rate 


District pair 1 


26.5 


30.6 


26.6 


28.7 


-4.1 


-2.1 


District pair 2 


37.4 


44.1 


41.4 


45.8 


-6.7 


-4.4 


District pair 3 


36.3 


19.9 


38.1 


12.6 


14.5 


25.5 


District pair 4 


30.9 


37.9 


31.8 


40.5 


-7.0 


-8.8 


District pair 5 


31.1 


32.5 


22.4 


29.1 


-1.4 


-6.7 


District pair 6 


36.2 


48.2 


39.4 


44.4 


-12.1 


-5.0 


District pair 7 


7.3 


7.6 


6.8 


10.9 


-0.3 


-4.1 


District pair 8 


31.9 


33.5 


37.2 


40.6 


-1.6 


-3.5 


District pair 9 


22.3 


28.1 


23.9 


21.8 


-5.8 


2.1 



Graduated Verification Sites 



Total Participation Rate 



District pair 1 


54.1 


62.5 


54.6 


62.5 


-8.4 


-7.9 


District pair 2 


76.9 


55.6 


73.7 


51.9 


21.4 


21.8 


District pair 3 


66.3 


68.8 


65.6 


63.9 


-2.5 


1.7 
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TABLE A.2 {continued) 



Mean Participation Rate“ 



Difference 



2002 1999 



Measure of Participation Pilot Comparison Pilot Comparison 2002 1999 



Free/Reduced-Price 
Participation Rate 



District pair 1 


24.6 


37.9 


22.0 


30.4 


-13.3 


-8.4 


District pair 2 


42.7 


33.7 


44.9 


31.3 


9.0 


13.7 


District pair 3 


17.3 


20.6 


18.0 


19.2 


-3.3 


-1.2 


Paid Participation Rate 


District pair 1 


29.5 


24.6 


32.6 


32.1 


4.8 


0.5 


District pair 2 


34.3 


21.9 


28.7 


20.6 


12.4 


8.1 


District pair 3 


49.0 


48.2 


47.6 


44.7 


0.8 


2.9 



Source: Internal memorandum from Paul Strasberg to John Burghardt and Phil Gleason, “NSLP Application/ 
Verification Pilot Projects: Analysis of the Effect on Participation,” September 23, 2003. 

“ Mean participation rate among all students attending district except those directly certified. 
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TABLE A.3 



ESTIMATED PARTICIPATION RATES BASED ON SURVEY DATA IN UP-ERONT DOCUMENTATION 
AND GRADUATED VERIEICATION SITES, BY DISTRICT AND PILOT STATUS 



Mean Participation Rate“ 



Measure of Participation Pilot Comparison Difference 



Up-Front Documentation Sites 



Total Participation Rate 



District pair 1 


66.4 


75.2 


-8.8 


District pair 2 


70.4 


73.7 


-3.3 


District pair 3 


75.7 


47.0 


28.7 


District pair 4 


44.3 


79.3 


-35.0 


District pair 5 


75.0 


65.1 


9.9 


District pair 6 


82.6 


78.5 


4.1 


District pair 7 


67.5 


81.3 


-13.8 


District pair 8 


57.2 


69.9 


-12.7 


District pair 9 


59.7 


89.4 


-29.7 


Eree/Reduced-Price Participation Rate 








District pair 1 


10.5 


28.3 


-17.8 


District pair 2 


31.5 


23.8 


7.7 


District pair 3 


18.3 


14.6 


3.7 


District pair 4 


10.2 


14.2 


-4.0 


District pair 5 


16.6 


10.4 


6.2 


District pair 6 


31.5 


22.0 


9.5 


District pair 7 


2.6 


18.5 


-15.9 


District pair 8 


4.8 


9.7 


-4.9 


District pair 9 


17.9 


28.7 


-10.8 


Paid Participation Rate 








District pair 1 


55.9 


47.0 


8.9 


District pair 2 


39.0 


49.7 


-10.7 


District pair 3 


57.1 


32.0 


25.1 


District pair 4 


33.9 


64.8 


-30.9 


District pair 5 


57.9 


54.7 


3.2 


District pair 6 


50.3 


56.5 


-6.2 


District pair 7 


64.9 


62.8 


2.1 


District pair 8 


52.3 


60.2 


-7.9 


District pair 9 


41.6 


60.8 


-19.2 


Graduated Verification Sites 


Total Participation Rate 








District pair 1 


74.8 


91.2 


-16.4 


District pair 2 


86.8 


72.6 


14.2 


District pair 3 


75.2 


86.3 


-11.1 


Eree/Reduced-Price Participation Rate 








District pair 1 


25.3 


45.5 


-20.2 


District pair 2 


473.5 


36.2 


7.3 


District pair 3 


17.4 


20.4 


-3.0 



A.14 






TABLE A.3 (continued) 





Mean Participation Rate“ 




Measure of Participation 


Pilot 


Comparison 


Difference 


Paid Participation Rate 








District pair 1 


49.0 


45.7 


3.3 


District pair 2 


43.0 


36.3 


6.7 


District pair 3 


57.8 


65.9 


-8.1 



“ Mean participation rate among all students attending district except those directly certified. 
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of the pilot on participation were not strongly influenced by the data source, although we address 
this question more carefully below. 

In Graduated Verification districts, parents also over-reported their children’s NSLP 
participation. At first glance, the over-reporting in Graduated Verification districts appears less 
prevalent than in Up-Front Documentation districts, since the difference between the 
participation rate based on survey data and that based on administrative data was smaller in 
absolute terms. In particular, the participation rate was 81.5 percent in Graduated Verification 
comparison districts according to parents, and 62.3 percent according to administrative data 
(Table A.l). This difference of about 19 percentage points was smaller than the absolute 
difference in Up-Front Documentation comparison districts of nearly 25 percentage points. 
However, this was primarily due to the fact that a larger proportion of students in Graduated 
Verification comparison districts got a school lunch on any given day (according to the 
administrative data), implying that there was less opportunity for over-reporting among parents 
in those districts. In fact, the rate of over-reporting in Graduated Verification comparison 
districts was 51 percent, compared with 47 percent in Up-Front Documentation comparison 
districts. 

Over-reporting of participation occurred less frequently in Graduated Verification pilot 
districts than in comparison districts. In particular, the absolute difference between the survey- 
based participation rate and administrative data-based participation rate was 19.2 percentage 
points in comparison districts compared with only 12.1 percentage points in pilot districts (Table 
A.l). Thus, while the survey data indicated a higher participation rate in comparison districts, the 
administrative data indicated a higher participation rate in pilot districts. This is a preliminary 
indication of a potential bias in the estimated impact of Graduated Verification on NSLP 
participation, but since this difference may have been driven by some district characteristic other 
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than the implementation of the pilot intervention, we need to look at the comparison of estimated 
impacts (after controlling for student/household characteristics) based on survey versus 
administrative data to get a better indication of potential bias. 

2. Estimated Impacts on NSLP Participation 

Table A.4 shows the estimated impacts of Up-Front Documentation and Graduated 
Verification on NSLP participation rates. In particular, for both survey and administrative data, 
the table shows the mean participation rate in comparison districts, the estimated impact in this 
rate in percentage points, and this impact measured as a percentage of the comparison district 
participation rate. 

Overall, estimated impacts based on survey data are similar to those obtained from the 
administrative data difference-in-differences model. In Up-Front Documentation districts, the 
estimated impact on the total participation rate was -8.0 percent according to survey data 
compared with -6.0 percent according to administrative data.^ The estimated impact of Up- 
Front Documentation on the free/reduced-price participation rate was negative in both cases, but 
the impact estimate was a bit different when based on survey data (-15.1 percent) compared with 
when based on administrative data (-5.6 percent). For both survey data and administrative data, 
however, Up-Front Documentation was estimated to lead to a 6.1 percent decrease in the paid 
participation rate. 

Despite the fact that the previous section described differential rates of over-reporting of 
participation in Graduated Verification pilot versus comparison districts, the estimated impact of 



^ In absolute terms, the estimated percentage point impact based on survey data is twice as 
large as the percentage point impact based on administrative data (-5.8 versus -2.9 percentage 
points). However, this difference is somewhat misleading because the comparison district 
participation rate is much lower according to administrative data than according to survey data. 
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TABLE A.4 



ESTIMATED IMPACTS ON PARTICIPATION RATES BASED ON SURVEY AND ADMINISTRATIVE DATA IN UP- 
FRONT DOCUMENTATION AND GRADUATED VERIFICATION SITES, BY PILOT STATUS^ 



Estimates Based on Administrative 
Estimates Based on Survey Data Data 



Measure of Participation 


Participation 
Rate in 
Comparison 
Districts 


Estimated 
Impact of 
Pilot 

(Percentage 

Points) 


Impact as 
Percentage 
of 

Participation 
Rate in 
Comparison 
Districts 


Participation 
Rate in 
Comparison 
Districts 


Estimated 
Impact of 
Pilot 

(Percentage 

Points) 


Impact as 
Percentage 
of 

Participation 
Rate in 
Comparison 
Districts 


Up-Front Documentation Sites 


Total Participation Rate 


72.6 


-5.8 


-8.0% 


48.1 


-2.9 


-6.0% 


Free/Reduced-Price 
Participation Rate 


18.5 


-2.8 


-15.1% 


16.0 


-0.9 


-5.6% 


Paid Participation Rate 


54.0 


-3.3 


-6.1% 


31.4 


-1.9 


-6.1% 


Graduated Verification Sites 


Total Participation Rate 


81.5 


-3.4 


-4.2% 


62.3 


-1.7 


-2.7% 


Free/Reduced-Price 
Participation Rate 


33.6 


-4.7 


-14.0% 


30.7 


-3.8 


-12.4% 


Paid Participation Rate 


47.9 


1.2 


2.5% 


31.6 


2.2 


7.0% 



Note: Estimates based on survey data are based on a regression model and are drawn from Chapter III, Tables III. 2 and III.7. 

Estimates based on administrative data are calculated based on a difference-in-differences model described in this 
appendix and using data from Tables A.2 and A.3. 

■‘Participation rate estimates and impact estimates are based on samples of students attending district except those directly 
certified. 
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Graduated Verification on the total participation rate was similar across the two data sources. 
Graduated Verification was estimated to lead to a 4.2 percent decline in the NSLP participation 
rate according to survey data and to a 2.7 percent decline according to administrative data. 

Unlike Up-Front Documentation, the estimated impact of Graduated Verification on the 
free/reduced-price participation rate was also very similar across the two data sources: -14.0 
percent for survey data and -12.4 percent for administrative data. However, Graduated 
Verification was estimated to lead to a small (2.5 percent) increase in paid participation 
according to survey data, compared with an somewhat larger increase of 7.0 percent according to 
administrative data. 

C. IMPLICATIONS OF THE ANALYSIS OF ADMINISTRATIVE DATA 

Based on data received from and preliminary analysis conducted by FNS, we used 
administrative data to address two questions: (1) How accurate were parents’ reports of their 
children’s levels of NSLP participation on the survey conducted for the Evaluation of NSLP 
Application/Verification Pilot Projects? (2) Can the data on NSLP participation reported by 
parents be used to estimate the impacts of Up-Lront Documentation and Graduated Verification 
on NSLP participation without fear of substantial bias? 

We determined that parent reports of NSLP participation by their children lead to higher 
estimates of the level of NSLP participation among students in the pilot and comparison districts 
than are revealed in the program administrative data. Parents are especially likely to over-report 
school meal participation by students who are not certified for free or reduced-price meals. 

Despite these differences in the levels of NSLP participation in the two data sources, the two 
sources provide similar estimates of pilot impacts. After important student/household 
characteristics were controlled for, either through a regression model or through a difference-in- 
differences model, the resulting estimates of the impacts of the pilots on participation based on 
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parent-reported survey data or on administrative data were similar. This finding increases our 
confidence that the estimated impacts on participation based on the survey data reported in 
Chapter III are accurate and reliable. 
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